Modify

Opened 13 months ago

Closed 13 months ago

Last modified 11 months ago

#20784 closed defect (fixed)

Can`t download data by Overpass Query when activate Arabic translation

Reported by: hubaishan Owned by: GerdP
Priority: major Milestone: 21.04
Component: Core Version: tested
Keywords: template_report Cc: Don-vip, stoecker, simon04

Description

What steps will reproduce the problem?

  1. I choose Arabic language from preferences.
  2. I ran an overpass query.
  3. I got this Error in Arabic

بلغ خادم OSM ' overpass-api.de ' عن طلب غير صحيح. رسالة خطأ (غير مترجمة): line 2: parse error: Operator expected, but "٦"found.

What is the expected result?

The same query ran OK in English interface

What happens instead?

Please provide any additional information below. Attach a screenshot if possible.

URL:https://josm.openstreetmap.de/svn/trunk
Repository:UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
Last:Changed Date: 2021-04-01 23:17:01 +0200 (Thu, 01 Apr 2021)
Build-Date:2021-04-01 21:46:03
Revision:17702
Relative:URL: ^/trunk

Identification: JOSM/1.5 (17702 ar) Windows 10 64-Bit
OS Build number: Windows 10 Pro 1909 (18363)
Memory Usage: ١٩٥٢ MB / ١٩٥٢ MB (١٧١٠ MB allocated, but free)
Java version: 1.8.0_231-b11, Oracle Corporation, Java HotSpot(TM) 64-Bit Server VM
Look and Feel: com.sun.java.swing.plaf.windows.WindowsLookAndFeel
Screen: \Display0 ١٩٢٠×١٠٨٠ (scaling ١٫٠٠×١٫٠٠)
Maximum Screen Size: 1920×1080
Best cursor sizes: 16×16→32×32, 32×32→32×32
System property file.encoding: Cp1256
System property sun.jnu.encoding: Cp1256

Plugins:
+ CommandLine (35705)
+ ElevationProfile (35640)
+ ImproveWay (29)
+ InfoMode (35543)
+ alignways (35640)
+ apache-commons (35524)
+ apache-http (35589)
+ auto_tools (73)
+ changeset-viewer (25)
+ continuosDownload (91)
+ contourmerge (v0.1.6)
+ editgpx (35562)
+ ejml (35458)
+ geotools (35458)
+ imagery_offset_db (35640)
+ jna (35662)
+ jts (35458)
+ mapwithai (1.7.1.4)
+ measurement (35640)
+ namemanager (35567)
+ pbf (35720)
+ photo_geotagging (35715)
+ reltoolbox (35640)
+ reverter (35688)
+ scripting (30798)
+ undelete (35640)
+ utilsplugin2 (35691)

Tagging presets:
+ https://josm.openstreetmap.de/josmfile?page=Presets/NewTags&zip=1
+ <josm.pref>\EasyPresets.xml
+ F:\Downloads\dhaka_presets.xml

Map paint styles:
- https://raw.githubusercontent.com/yopaseopor/indoormap/master/indoormap-style.mapcss
- %UserProfile%shan\AppData\Roaming\JOSM\Hubaishan.mapcss
- "<josm.pref>\Hubaishan.mapcss"
- https://josm.openstreetmap.de/josmfile?page=Styles/AddressValidator&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/Admin_Boundaries&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/AdvertisingStyle&zip=1
- https://josm.openstreetmap.de/josmfile?page=Styles/Bench&zip=1
+ D:\My\Documents\0GIS\hubaihsanStyle.mapcss
+ https://josm.openstreetmap.de/josmfile?page=Styles/MapWithAI&zip=1

Last errors/warnings:
- ٠٠٠٠٨٫١٠٨ W: PluginException: : org.openstreetmap.josm.plugins.PluginException: ملف jar  غير صالح ' <josm.pref>\plugins\RoadSigns.jar.new '
- ٠٠٠٠٨٫١٠٨ W: أخفق مسح الملف ' RoadSigns.jar.new ' للحصول على معلومات الإضافة. تخطي.
- ٠٠٠١٤٫٣٥٥ W: فشل تثبيت الإضافة ' <josm.pref>\plugins\RoadSigns.jar ' من ملف التنزيل المؤقت ' <josm.pref>\plugins\RoadSigns.jar.new '. error in opening zip file: java.util.zip.ZipException: error in opening zip file
- ٠٠٠٢٠٫١٥٩ W: فشل تحميل أنماط Mappaint من ' %UserProfile%shan\AppData\Roaming\JOSM\Hubaishan.mapcss '. السبب هو: java.nio.file.NoSuchFileException: %UserProfile%shan\AppData\Roaming\JOSM\Hubaishan.mapcss
- ٠٠٠٢٠٫١٦٠ E: java.nio.file.NoSuchFileException: %UserProfile%shan\AppData\Roaming\JOSM\Hubaishan.mapcss
- ٠٠٠٢٠٫١٦١ W: فشل تحميل أنماط Mappaint من ' "<josm.pref>\Hubaishan.mapcss" '. السبب هو: java.nio.file.InvalidPathException: Illegal char <"> at index 0: "<josm.pref>\Hubaishan.mapcss"
- ٠٠٠٢٠٫١٦١ E: java.nio.file.InvalidPathException: Illegal char <"> at index 0: "<josm.pref>\Hubaishan.mapcss"
- ٠٠٠٣٢٫٠٩٢ E: فشل في تحديد موقع الصورة ' MapWithAI '
- ٠٠٠٥٨٫١٢١ E: org.openstreetmap.josm.io.OsmApiException: ResponseCode=400, Error Header=<line 2: parse error: Operator expected, but &quot;٦&quot;found.>, Error Body=<<?xml version="1.0" encoding="UTF-8"?>
- ٠٠٠٥٨٫١٦٤ E: طلب سيّء - <html>أبلغ خادم OSM ' overpass-api.de ' عن طلب غير صحيح. <br><br> رسالة خطأ (غير مترجمة): line 2: parse error: Operator expected, but &quot;٦&quot;found. </html>

Attachments (2)

Indic numbers.reg (272 bytes) - added by hubaishan 13 months ago.
change locale to Arabic-indic numbers
revert Indic numbers.reg (272 bytes) - added by hubaishan 13 months ago.
Revert changing locale to Arabic-Indic numbers

Download all attachments as: .zip

Change History (42)

comment:1 Changed 13 months ago by GerdP

Please attach the overpass query for arabic and for english. Is the arabic query working when you use it here? http://overpass-turbo.eu/

comment:2 Changed 13 months ago by hubaishan

All queries that success in English interface failed in Arabic interface

comment:3 Changed 13 months ago by GerdP

I cannot reproduce this so far. I started r17702 with option --language=ar
I used overpass wizzard to download landuse=forest in a small area and the generated query looks good and the executed query returns the expected result.
Do you queries contain arabic text?

comment:4 Changed 13 months ago by hubaishan

No.
this is a sample:

[out:xml][timeout:90];

{{geocodeArea:Saudi Arabia}}->.searchArea;
(
  nwr["place"="town"](area.searchArea);
);
(._;>;);
out meta;

comment:5 Changed 13 months ago by GerdP

Hmm, this query also works for me.

comment:6 Changed 13 months ago by hubaishan

Maybe a plugin bug? how can I disable all pulgins?

Last edited 13 months ago by hubaishan (previous) (diff)

comment:7 Changed 13 months ago by GerdP

start JOSM with option --skip-plugins

comment:8 Changed 13 months ago by GerdP

I've noticed that your status report shows arabic characters in the line with memory usage, mine doesn't. Don't know how this is possible, but if that also happens with the value "90" in "timeout:90" it would explain the error message.

comment:9 in reply to:  8 Changed 13 months ago by hubaishan

Replying to GerdP:

I've noticed that your status report shows arabic characters in the line with memory usage, mine doesn't. Don't know how this is possible, but if that also happens with the value "90" in "timeout:90" it would explain the error message.

Your notice is good.
The error is when choosing "Arabic(Saudi Arabia)" in Regional Setting in Windows. "Arabic(Saudi Arabia)" uses Arabic–Hindu numerals see https://en.wikipedia.org/wiki/Eastern_Arabic_numerals. When change Regional Setting in Windows to "Arabic(Algeria)" which uses Arabic numbers I got no error.

So JOSM must not translate any data sent to servers.

comment:10 Changed 13 months ago by GerdP

I guess all places where e.g. String.format("%d", count) is used to convert a number to a readable string are involved. I assume they all should use a locale, like String.format(Locale.ENGLISH, "%d", count) ?

Last edited 13 months ago by GerdP (previous) (diff)

comment:11 Changed 13 months ago by GerdP

Cc: Don-vip stoecker simon04 added

comment:12 Changed 13 months ago by stoecker

Don't use String.format() at all, but Integer.toString() or similar safe functions for these places generating non-user visible output?

comment:13 Changed 13 months ago by GerdP

Owner: changed from team to GerdP
Status: newassigned

Yes, probably. It's just not so easy to find out where. We have code in OverpassDownloadReader that parses the query and replaces some strings like geocodeArea:Saudi Arabia to area(3600307584).
Those probably cause the trouble, so I'll start with that class.

comment:14 Changed 13 months ago by GerdP

Resolution: fixed
Status: assignedclosed

In 17811/josm:

fix #20784: Can`t download data by Overpass Query when activate Arabic translation

  • use Locale.English when converting numbers to readable strings

comment:15 Changed 13 months ago by GerdP

@Dirk:
Which do you prefer?

  return "area(" + Long.toString(osmId.getUniqueId() + idOffset.get(osmId.getType())) + ")";

or maybe

  return String.format("area(%s)", Long.toString(osmId.getUniqueId() + idOffset.get(osmId.getType())));

is better than

  return String.format(Locale.ENGLISH, "area(%d)", osmId.getUniqueId() + idOffset.get(osmId.getType()));

comment:16 Changed 13 months ago by stoecker

If the current version works it should be ok. It's only that if in the future some java developer idiot decides that it is a good idea to change English locale...

My "feeling" would be that the first is safest. ;-)

comment:17 Changed 13 months ago by GerdP

I prefer to add the locale as it makes clearer why the code was changed. I think the code in ShowStatusReportAction should also be changed because it doesn't help us when memory values are reported with Arabic–Hindu. New ticket?

comment:18 in reply to:  17 Changed 13 months ago by stoecker

Replying to GerdP:

New ticket?

To be perfect: yes. Or you can simply fix it without a ticket ;-)

comment:19 Changed 13 months ago by GerdP

I think the status report should contain a hint about this "Regional Setting in Windows". Not sure how to report that. I'll open a new ticket.

comment:20 Changed 13 months ago by GerdP

see #20785

comment:21 Changed 13 months ago by GerdP

@hubaishan: I've not tested my fix because I don't dare to change my Windows system to a language that I can't read at all.
I still don't understand why it worked when you selected English language in JOSM, so maybe there is more to this.

comment:22 Changed 13 months ago by skyper

As far as I understand the situation, the problem is the underlying setting for numbers. Is this setting adjusted when the general language is changed in JOSM? I do not find any settings in JOSM for number format, e.g. how to use German but with the English notation for numbers?

comment:23 Changed 13 months ago by GerdP

There is code in I18n.initializeNumberingFormat() which seems to address this problem, see #18856

comment:24 Changed 13 months ago by Don-vip

Probably a side effect of r16109, thanks for fixing it.

comment:25 Changed 13 months ago by GerdP

Doesn't this ticket show that the changes in r16109 are not doing what they should?
Why didn't I see the problem with parameter --language=ar ?

comment:26 Changed 13 months ago by Don-vip

Because it depends on the country too. The changes do that they're meant to: prefer eastern arabic digits over western arabic ones in countries where eastern digits are the official system. Please read #18856 comments for context. Of course it's meant for UI only, not for APIs or I/O where western digits must be forced.

comment:27 in reply to:  22 Changed 13 months ago by Don-vip

Replying to skyper:

German but with the English notation for numbers?

There is no such thing. English and German use the same numbering system.

comment:28 Changed 13 months ago by GerdP

I think skyper means the decimal separators 1.234 vs 1,234

comment:29 in reply to:  26 Changed 13 months ago by GerdP

Replying to Don-vip:

Because it depends on the country too. The changes do that they're meant to: prefer eastern arabic digits over western arabic ones in countries where eastern digits are the official system. Please read #18856 comments for context. Of course it's meant for UI only, not for APIs or I/O where western digits must be forced.

Ah, OK, I misread comment:9 and thought there are two different possible configs for Saudi Arabia.

comment:30 in reply to:  21 Changed 13 months ago by abdullah

Replying to GerdP:

@hubaishan: I've not tested my fix because I don't dare to change my Windows system to a language that I can't read at all.
I still don't understand why it worked when you selected English language in JOSM, so maybe there is more to this.

You don't need to change the system language to test this, you just need to change the number format as in the following picture:
https://i.imgur.com/KOvDfSM.png

comment:31 Changed 13 months ago by abdullah

I also tested the problem and found that it is related to that setting shown in the picture, regardless of the country chosen for the format

comment:32 in reply to:  16 Changed 13 months ago by simon04

Replying to stoecker:

If the current version works it should be ok. It's only that if in the future some java developer idiot decides that it is a good idea to change English locale...

My "feeling" would be that the first is safest. ;-)

Java also provides Locale.ROOThttps://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/Locale.html#ROOT

This is regarded as the base locale of all locales, and is used as the language/country neutral locale for the locale sensitive operations.

Changed 13 months ago by hubaishan

Attachment: Indic numbers.reg added

change locale to Arabic-indic numbers

Changed 13 months ago by hubaishan

Attachment: revert Indic numbers.reg added

Revert changing locale to Arabic-Indic numbers

comment:33 Changed 13 months ago by hubaishan

With these reg files you can test the bug without changing the locale of Windows

comment:34 Changed 13 months ago by GerdP

I tried this together with --language=ar and I see the Arabic-Indic numbers in Windows and also in JOSM, but I cannot reproduce the overpass problem. In the debugger the query is displayed with Arabic-Indic numbers but it seems that the actual query is still correct, the registry change just seems to change the rendering in Windows, not the result of the String.format() routine. I guess this only happens with a different locale.
Maybe you can try the latest version? https://josm.openstreetmap.de/josm-latest.jar

comment:35 Changed 13 months ago by GerdP

Ah, finally: With JRE options -Duser.country=SA -Duser.language=ar and JOSM option --language=ar I can reproduce the problem and the changes in r17810 really seem to fix it.

Last edited 13 months ago by GerdP (previous) (diff)

comment:36 Changed 13 months ago by hubaishan

Thank you. It is Fixed in r17810

comment:37 in reply to:  28 Changed 13 months ago by skyper

Replying to GerdP:

I think skyper means the decimal separators 1.234 vs 1,234

Yes, the separators comma and dot which are switched and the optional leading zero in cases of values between zero and one.

I usually choose some Irish settings as I want to have the correct currency but only use English numbering format as I remember several issue with reformatting scientific data in advance of analyzing.

Last edited 13 months ago by skyper (previous) (diff)

comment:38 Changed 13 months ago by GerdP

@skyper: I don't understand where this is an issue. Please open a new ticket.

comment:39 Changed 13 months ago by skyper

No problem for me, as I use English all the way and I do not have to admin a machine of any other JOSM user. Offering currencies and physical units, similar to colors, might be worth a new ticket but I have to think about it and this probably depends more on the edit region than on user's settings.

comment:40 Changed 11 months ago by Don-vip

Milestone: 21.04

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The owner will remain GerdP.
as The resolution will be set.
The resolution will be deleted.

Add Comment


E-mail address and name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.