#20784 closed defect (fixed)
Can`t download data by Overpass Query when activate Arabic translation
| Reported by: | hubaishan | Owned by: | GerdP |
|---|---|---|---|
| Priority: | major | Milestone: | 21.04 |
| Component: | Core | Version: | tested |
| Keywords: | template_report | Cc: | Don-vip, stoecker, simon04 |
Description
What steps will reproduce the problem?
- I choose Arabic language from preferences.
- I ran an overpass query.
- I got this Error in Arabic
بلغ خادم OSM ' overpass-api.de ' عن طلب غير صحيح. رسالة خطأ (غير مترجمة): line 2: parse error: Operator expected, but "٦"found.
What is the expected result?
The same query ran OK in English interface
What happens instead?
Please provide any additional information below. Attach a screenshot if possible.
URL:https://josm.openstreetmap.de/svn/trunk Repository:UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b Last:Changed Date: 2021-04-01 23:17:01 +0200 (Thu, 01 Apr 2021) Build-Date:2021-04-01 21:46:03 Revision:17702 Relative:URL: ^/trunk Identification: JOSM/1.5 (17702 ar) Windows 10 64-Bit OS Build number: Windows 10 Pro 1909 (18363) Memory Usage: ١٩٥٢ MB / ١٩٥٢ MB (١٧١٠ MB allocated, but free) Java version: 1.8.0_231-b11, Oracle Corporation, Java HotSpot(TM) 64-Bit Server VM Look and Feel: com.sun.java.swing.plaf.windows.WindowsLookAndFeel Screen: \Display0 ١٩٢٠×١٠٨٠ (scaling ١٫٠٠×١٫٠٠) Maximum Screen Size: 1920×1080 Best cursor sizes: 16×16→32×32, 32×32→32×32 System property file.encoding: Cp1256 System property sun.jnu.encoding: Cp1256 Plugins: + CommandLine (35705) + ElevationProfile (35640) + ImproveWay (29) + InfoMode (35543) + alignways (35640) + apache-commons (35524) + apache-http (35589) + auto_tools (73) + changeset-viewer (25) + continuosDownload (91) + contourmerge (v0.1.6) + editgpx (35562) + ejml (35458) + geotools (35458) + imagery_offset_db (35640) + jna (35662) + jts (35458) + mapwithai (1.7.1.4) + measurement (35640) + namemanager (35567) + pbf (35720) + photo_geotagging (35715) + reltoolbox (35640) + reverter (35688) + scripting (30798) + undelete (35640) + utilsplugin2 (35691) Tagging presets: + https://josm.openstreetmap.de/josmfile?page=Presets/NewTags&zip=1 + <josm.pref>\EasyPresets.xml + F:\Downloads\dhaka_presets.xml Map paint styles: - https://raw.githubusercontent.com/yopaseopor/indoormap/master/indoormap-style.mapcss - %UserProfile%shan\AppData\Roaming\JOSM\Hubaishan.mapcss - "<josm.pref>\Hubaishan.mapcss" - https://josm.openstreetmap.de/josmfile?page=Styles/AddressValidator&zip=1 - https://josm.openstreetmap.de/josmfile?page=Styles/Admin_Boundaries&zip=1 - https://josm.openstreetmap.de/josmfile?page=Styles/AdvertisingStyle&zip=1 - https://josm.openstreetmap.de/josmfile?page=Styles/Bench&zip=1 + D:\My\Documents\0GIS\hubaihsanStyle.mapcss + https://josm.openstreetmap.de/josmfile?page=Styles/MapWithAI&zip=1 Last errors/warnings: - ٠٠٠٠٨٫١٠٨ W: PluginException: : org.openstreetmap.josm.plugins.PluginException: ملف jar غير صالح ' <josm.pref>\plugins\RoadSigns.jar.new ' - ٠٠٠٠٨٫١٠٨ W: أخفق مسح الملف ' RoadSigns.jar.new ' للحصول على معلومات الإضافة. تخطي. - ٠٠٠١٤٫٣٥٥ W: فشل تثبيت الإضافة ' <josm.pref>\plugins\RoadSigns.jar ' من ملف التنزيل المؤقت ' <josm.pref>\plugins\RoadSigns.jar.new '. error in opening zip file: java.util.zip.ZipException: error in opening zip file - ٠٠٠٢٠٫١٥٩ W: فشل تحميل أنماط Mappaint من ' %UserProfile%shan\AppData\Roaming\JOSM\Hubaishan.mapcss '. السبب هو: java.nio.file.NoSuchFileException: %UserProfile%shan\AppData\Roaming\JOSM\Hubaishan.mapcss - ٠٠٠٢٠٫١٦٠ E: java.nio.file.NoSuchFileException: %UserProfile%shan\AppData\Roaming\JOSM\Hubaishan.mapcss - ٠٠٠٢٠٫١٦١ W: فشل تحميل أنماط Mappaint من ' "<josm.pref>\Hubaishan.mapcss" '. السبب هو: java.nio.file.InvalidPathException: Illegal char <"> at index 0: "<josm.pref>\Hubaishan.mapcss" - ٠٠٠٢٠٫١٦١ E: java.nio.file.InvalidPathException: Illegal char <"> at index 0: "<josm.pref>\Hubaishan.mapcss" - ٠٠٠٣٢٫٠٩٢ E: فشل في تحديد موقع الصورة ' MapWithAI ' - ٠٠٠٥٨٫١٢١ E: org.openstreetmap.josm.io.OsmApiException: ResponseCode=400, Error Header=<line 2: parse error: Operator expected, but "٦"found.>, Error Body=<<?xml version="1.0" encoding="UTF-8"?> - ٠٠٠٥٨٫١٦٤ E: طلب سيّء - <html>أبلغ خادم OSM ' overpass-api.de ' عن طلب غير صحيح. <br><br> رسالة خطأ (غير مترجمة): line 2: parse error: Operator expected, but "٦"found. </html>
Attachments (2)
Change History (42)
comment:1 by , 4 years ago
comment:3 by , 4 years ago
I cannot reproduce this so far. I started r17702 with option --language=ar
I used overpass wizzard to download landuse=forest in a small area and the generated query looks good and the executed query returns the expected result.
Do you queries contain arabic text?
comment:4 by , 4 years ago
No.
this is a sample:
[out:xml][timeout:90];
{{geocodeArea:Saudi Arabia}}->.searchArea;
(
nwr["place"="town"](area.searchArea);
);
(._;>;);
out meta;
follow-up: 9 comment:8 by , 4 years ago
I've noticed that your status report shows arabic characters in the line with memory usage, mine doesn't. Don't know how this is possible, but if that also happens with the value "90" in "timeout:90" it would explain the error message.
comment:9 by , 4 years ago
Replying to GerdP:
I've noticed that your status report shows arabic characters in the line with memory usage, mine doesn't. Don't know how this is possible, but if that also happens with the value "90" in "timeout:90" it would explain the error message.
Your notice is good.
The error is when choosing "Arabic(Saudi Arabia)" in Regional Setting in Windows. "Arabic(Saudi Arabia)" uses Arabic–Hindu numerals see https://en.wikipedia.org/wiki/Eastern_Arabic_numerals. When change Regional Setting in Windows to "Arabic(Algeria)" which uses Arabic numbers I got no error.
So JOSM must not translate any data sent to servers.
comment:10 by , 4 years ago
I guess all places where e.g. String.format("%d", count) is used to convert a number to a readable string are involved. I assume they all should use a locale, like String.format(Locale.ENGLISH, "%d", count) ?
comment:11 by , 4 years ago
| Cc: | added |
|---|
comment:12 by , 4 years ago
Don't use String.format() at all, but Integer.toString() or similar safe functions for these places generating non-user visible output?
comment:13 by , 4 years ago
| Owner: | changed from to |
|---|---|
| Status: | new → assigned |
Yes, probably. It's just not so easy to find out where. We have code in OverpassDownloadReader that parses the query and replaces some strings like geocodeArea:Saudi Arabia to area(3600307584).
Those probably cause the trouble, so I'll start with that class.
comment:15 by , 4 years ago
@Dirk:
Which do you prefer?
return "area(" + Long.toString(osmId.getUniqueId() + idOffset.get(osmId.getType())) + ")";
or maybe
return String.format("area(%s)", Long.toString(osmId.getUniqueId() + idOffset.get(osmId.getType())));
is better than
return String.format(Locale.ENGLISH, "area(%d)", osmId.getUniqueId() + idOffset.get(osmId.getType()));
follow-up: 32 comment:16 by , 4 years ago
If the current version works it should be ok. It's only that if in the future some java developer idiot decides that it is a good idea to change English locale...
My "feeling" would be that the first is safest. ;-)
follow-up: 18 comment:17 by , 4 years ago
I prefer to add the locale as it makes clearer why the code was changed. I think the code in ShowStatusReportAction should also be changed because it doesn't help us when memory values are reported with Arabic–Hindu. New ticket?
comment:18 by , 4 years ago
comment:19 by , 4 years ago
I think the status report should contain a hint about this "Regional Setting in Windows". Not sure how to report that. I'll open a new ticket.
follow-up: 30 comment:21 by , 4 years ago
@hubaishan: I've not tested my fix because I don't dare to change my Windows system to a language that I can't read at all.
I still don't understand why it worked when you selected English language in JOSM, so maybe there is more to this.
follow-up: 27 comment:22 by , 4 years ago
As far as I understand the situation, the problem is the underlying setting for numbers. Is this setting adjusted when the general language is changed in JOSM? I do not find any settings in JOSM for number format, e.g. how to use German but with the English notation for numbers?
comment:23 by , 4 years ago
There is code in I18n.initializeNumberingFormat() which seems to address this problem, see #18856
comment:25 by , 4 years ago
Doesn't this ticket show that the changes in r16109 are not doing what they should?
Why didn't I see the problem with parameter --language=ar ?
follow-up: 29 comment:26 by , 4 years ago
Because it depends on the country too. The changes do that they're meant to: prefer eastern arabic digits over western arabic ones in countries where eastern digits are the official system. Please read #18856 comments for context. Of course it's meant for UI only, not for APIs or I/O where western digits must be forced.
comment:27 by , 4 years ago
Replying to skyper:
German but with the English notation for numbers?
There is no such thing. English and German use the same numbering system.
follow-up: 37 comment:28 by , 4 years ago
I think skyper means the decimal separators 1.234 vs 1,234
comment:29 by , 4 years ago
Replying to Don-vip:
Because it depends on the country too. The changes do that they're meant to: prefer eastern arabic digits over western arabic ones in countries where eastern digits are the official system. Please read #18856 comments for context. Of course it's meant for UI only, not for APIs or I/O where western digits must be forced.
Ah, OK, I misread comment:9 and thought there are two different possible configs for Saudi Arabia.
comment:30 by , 4 years ago
Replying to GerdP:
@hubaishan: I've not tested my fix because I don't dare to change my Windows system to a language that I can't read at all.
I still don't understand why it worked when you selected English language in JOSM, so maybe there is more to this.
You don't need to change the system language to test this, you just need to change the number format as in the following picture:
https://i.imgur.com/KOvDfSM.png
comment:31 by , 4 years ago
I also tested the problem and found that it is related to that setting shown in the picture, regardless of the country chosen for the format
comment:32 by , 4 years ago
Replying to stoecker:
If the current version works it should be ok. It's only that if in the future some java developer idiot decides that it is a good idea to change English locale...
My "feeling" would be that the first is safest. ;-)
Java also provides Locale.ROOT – https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/Locale.html#ROOT
This is regarded as the base locale of all locales, and is used as the language/country neutral locale for the locale sensitive operations.
by , 4 years ago
| Attachment: | revert Indic numbers.reg added |
|---|
Revert changing locale to Arabic-Indic numbers
comment:33 by , 4 years ago
With these reg files you can test the bug without changing the locale of Windows
comment:34 by , 4 years ago
I tried this together with --language=ar and I see the Arabic-Indic numbers in Windows and also in JOSM, but I cannot reproduce the overpass problem. In the debugger the query is displayed with Arabic-Indic numbers but it seems that the actual query is still correct, the registry change just seems to change the rendering in Windows, not the result of the String.format() routine. I guess this only happens with a different locale.
Maybe you can try the latest version? https://josm.openstreetmap.de/josm-latest.jar
comment:35 by , 4 years ago
Ah, finally: With JRE options -Duser.country=SA -Duser.language=ar and JOSM option --language=ar I can reproduce the problem and the changes in r17810 really seem to fix it.
comment:37 by , 4 years ago
Replying to GerdP:
I think skyper means the decimal separators 1.234 vs 1,234
Yes, the separators comma and dot which are switched and the optional leading zero in cases of values between zero and one.
I usually choose some Irish settings as I want to have the correct currency but only use English numbering format as I remember several issue with reformatting scientific data in advance of analyzing.
comment:38 by , 4 years ago
@skyper: I don't understand where this is an issue. Please open a new ticket.
comment:39 by , 4 years ago
No problem for me, as I use English all the way and I do not have to admin a machine of any other JOSM user. Offering currencies and physical units, similar to colors, might be worth a new ticket but I have to think about it and this probably depends more on the edit region than on user's settings.
comment:40 by , 4 years ago
| Milestone: | → 21.04 |
|---|
Please attach the overpass query for arabic and for english. Is the arabic query working when you use it here? http://overpass-turbo.eu/