Context Navigation

Modify ↓

#19624 closed enhancement (fixed)

[PATCH] Support reading line-delimited geojson (RFC 7464)

Reported by:	taylor.smock	Owned by:	simon04
Priority:	normal	Milestone:	20.08
Component:	Core geojson	Version:
Keywords:	rfc 7464, maproulette	Cc:

Description

Maproulette uses RFC 7464. It would be useful to be able to check a file prior to uploading to Maproulette/check that the file is properly formatted. This is (currently) just a reader, although a writer implementation should also be possible.

This patch does the following:

Checks to see if the first byte of the file is the record separator as defined in RFC 7464 (0x1E)
If so, it reads the file line-by-line and creates a new parser for each line

What this patch does not do:

Ensure that the ending character for each line is 0x0A as specified by the RFC
Check for the original method Maproulette used for line-by-line geojson (^{.*}$)

What may need to be changed:

I may need to move the logic out to a separate function, for readability

Attachments (2)

19624.patch (3.2 KB ) - added by taylor.smock 4 years ago.: RFC 7464 reading implementation
19624.1.patch (13.9 KB ) - added by taylor.smock 4 years ago.: Add tests, fix tests (due to input streams not supporting marks)

Download all attachments as: .zip

Change History (16)

by taylor.smock, 4 years ago

Attachment:	19624.patch added

RFC 7464 reading implementation

comment:1 by taylor.smock, 4 years ago

Is there any interest for this patch?

comment:2 by stoecker, 4 years ago

Probably, if I understand it :-)

follow-up: 4 comment:3 by simon04, 4 years ago

Owner:	changed from team to simon04
Status:	new → assigned

Relevant Maproulette commit: https://github.com/maproulette/maproulette2/commit/015c48094dbfae5d7a1d875b148d9e41fc51f3cc

The GeoJSONReaderTest unit tests fail:

System property josm.test.data is not set, using 'test/data'
2020-08-22 18:57:18.019 SEVERE: java.io.IOException: mark/reset not supported
java.io.IOException: mark/reset not supported
	at java.base/java.io.InputStream.reset(InputStream.java:655)
	at org.openstreetmap.josm.io.GeoJSONReader.isLineDelimited(GeoJSONReader.java:372)
	at org.openstreetmap.josm.io.GeoJSONReader.doParseDataSet(GeoJSONReader.java:382)
	at org.openstreetmap.josm.io.GeoJSONReaderTest.testReadGeoJson(GeoJSONReaderTest.java:47)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at mockit.integration.junit4.JUnit4TestRunnerDecorator.executeTestMethod(JUnit4TestRunnerDecorator.java:157)
	at mockit.integration.junit4.JUnit4TestRunnerDecorator.invokeExplosively(JUnit4TestRunnerDecorator.java:71)
	at mockit.integration.junit4.FakeFrameworkMethod.invokeExplosively(FakeFrameworkMethod.java:29)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.openstreetmap.josm.testutils.JOSMTestRules$TimeoutThread.run(JOSMTestRules.java:777)


org.openstreetmap.josm.io.IllegalDataException: javax.json.stream.JsonParsingException: Expected EOF token, but got COLON

	at org.openstreetmap.josm.io.GeoJSONReader.doParseDataSet(GeoJSONReader.java:408)
	at org.openstreetmap.josm.io.GeoJSONReaderTest.testReadGeoJson(GeoJSONReaderTest.java:47)
	at org.openstreetmap.josm.testutils.JOSMTestRules$TimeoutThread.run(JOSMTestRules.java:777)
Caused by: javax.json.stream.JsonParsingException: Expected EOF token, but got COLON
	at org.glassfish.json.JsonParserImpl.hasNext(JsonParserImpl.java:361)
	at org.openstreetmap.josm.io.GeoJSONReader.parse(GeoJSONReader.java:73)
	at org.openstreetmap.josm.io.GeoJSONReader.doParseDataSet(GeoJSONReader.java:406)
	... 2 more

For the new feature, I'd like to have a unit test as well…

in reply to: 3 comment:4 by taylor.smock, 4 years ago

Replying to simon04:

Relevant Maproulette commit: https://github.com/maproulette/maproulette2/commit/015c48094dbfae5d7a1d875b148d9e41fc51f3cc

The GeoJSONReaderTest unit tests fail:
For the new feature, I'd like to have a unit test as well…

I'll try to get a unit test written this week (I didn't take too much time on the initial patch, since I didn't want to spend time if there was no interest -- I just got it "good enough" for what I needed at the time).