Modify

Opened 7 weeks ago

Closed 4 weeks ago

Last modified 3 weeks ago

#19624 closed enhancement (fixed)

[PATCH] Support reading line-delimited geojson (RFC 7464)

Reported by: taylor.smock Owned by: simon04
Priority: normal Milestone: 20.08
Component: Core geojson Version:
Keywords: rfc 7464, maproulette Cc:

Description

Maproulette uses RFC 7464. It would be useful to be able to check a file prior to uploading to Maproulette/check that the file is properly formatted. This is (currently) just a reader, although a writer implementation should also be possible.

This patch does the following:

  1. Checks to see if the first byte of the file is the record separator as defined in RFC 7464 (0x1E)
  2. If so, it reads the file line-by-line and creates a new parser for each line

What this patch does not do:

  • Ensure that the ending character for each line is 0x0A as specified by the RFC
  • Check for the original method Maproulette used for line-by-line geojson (^{.*}$)

What may need to be changed:

  • I may need to move the logic out to a separate function, for readability

Attachments (2)

19624.patch (3.2 KB) - added by taylor.smock 7 weeks ago.
RFC 7464 reading implementation
19624.1.patch (13.9 KB) - added by taylor.smock 4 weeks ago.
Add tests, fix tests (due to input streams not supporting marks)

Download all attachments as: .zip

Change History (15)

Changed 7 weeks ago by taylor.smock

Attachment: 19624.patch added

RFC 7464 reading implementation

comment:1 Changed 4 weeks ago by taylor.smock

Is there any interest for this patch?

comment:2 Changed 4 weeks ago by stoecker

Probably, if I understand it :-)

comment:3 Changed 4 weeks ago by simon04

Owner: changed from team to simon04
Status: newassigned

Relevant Maproulette commit: https://github.com/maproulette/maproulette2/commit/015c48094dbfae5d7a1d875b148d9e41fc51f3cc

The GeoJSONReaderTest unit tests fail:

System property josm.test.data is not set, using 'test/data'
2020-08-22 18:57:18.019 SEVERE: java.io.IOException: mark/reset not supported
java.io.IOException: mark/reset not supported
	at java.base/java.io.InputStream.reset(InputStream.java:655)
	at org.openstreetmap.josm.io.GeoJSONReader.isLineDelimited(GeoJSONReader.java:372)
	at org.openstreetmap.josm.io.GeoJSONReader.doParseDataSet(GeoJSONReader.java:382)
	at org.openstreetmap.josm.io.GeoJSONReaderTest.testReadGeoJson(GeoJSONReaderTest.java:47)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at mockit.integration.junit4.JUnit4TestRunnerDecorator.executeTestMethod(JUnit4TestRunnerDecorator.java:157)
	at mockit.integration.junit4.JUnit4TestRunnerDecorator.invokeExplosively(JUnit4TestRunnerDecorator.java:71)
	at mockit.integration.junit4.FakeFrameworkMethod.invokeExplosively(FakeFrameworkMethod.java:29)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.openstreetmap.josm.testutils.JOSMTestRules$TimeoutThread.run(JOSMTestRules.java:777)


org.openstreetmap.josm.io.IllegalDataException: javax.json.stream.JsonParsingException: Expected EOF token, but got COLON

	at org.openstreetmap.josm.io.GeoJSONReader.doParseDataSet(GeoJSONReader.java:408)
	at org.openstreetmap.josm.io.GeoJSONReaderTest.testReadGeoJson(GeoJSONReaderTest.java:47)
	at org.openstreetmap.josm.testutils.JOSMTestRules$TimeoutThread.run(JOSMTestRules.java:777)
Caused by: javax.json.stream.JsonParsingException: Expected EOF token, but got COLON
	at org.glassfish.json.JsonParserImpl.hasNext(JsonParserImpl.java:361)
	at org.openstreetmap.josm.io.GeoJSONReader.parse(GeoJSONReader.java:73)
	at org.openstreetmap.josm.io.GeoJSONReader.doParseDataSet(GeoJSONReader.java:406)
	... 2 more

For the new feature, I'd like to have a unit test as well…

comment:4 in reply to:  3 Changed 4 weeks ago by taylor.smock

Replying to simon04:

Relevant Maproulette commit: https://github.com/maproulette/maproulette2/commit/015c48094dbfae5d7a1d875b148d9e41fc51f3cc

The GeoJSONReaderTest unit tests fail:
For the new feature, I'd like to have a unit test as well…

I'll try to get a unit test written this week (I didn't take too much time on the initial patch, since I didn't want to spend time if there was no interest -- I just got it "good enough" for what I needed at the time).

Changed 4 weeks ago by taylor.smock

Attachment: 19624.1.patch added

Add tests, fix tests (due to input streams not supporting marks)

comment:5 Changed 4 weeks ago by simon04

Resolution: fixed
Status: assignedclosed

In 16933/josm:

fix #19624 - Support reading line-delimited GeoJSON (RFC 7464; patch by taylor.smock)

comment:6 Changed 4 weeks ago by simon04

In 16934/josm:

see #19624 - Extract GeoJSONReader.parse(InputStream), use try-with-resources

comment:7 Changed 4 weeks ago by simon04

Milestone: 20.08

Thank you!

comment:8 Changed 4 weeks ago by simon04

In 16935/josm:

see #19624 - Fix accidentally disabled GeoJSONReaderTest

comment:9 in reply to:  8 Changed 4 weeks ago by taylor.smock

Replying to simon04:

In 16935/josm:

see #19624 - Fix accidentally disabled GeoJSONReaderTest

Oops. I guess I forgot to add the @Test back. :(

comment:10 Changed 4 weeks ago by simon04

No worries!

comment:11 Changed 3 weeks ago by simon04

In 16954/josm:

see #19624 - specify UTF-8 (Java Warnings)

comment:12 Changed 3 weeks ago by simon04

In 16955/josm:

see #19624 - use Utils.strip

comment:13 Changed 3 weeks ago by simon04

In 16956/josm:

see #19624 - use try-with-resources (PMD)

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The owner will remain simon04.
as The resolution will be set.
The resolution will be deleted.

Add Comment


E-mail address and name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.