Context Navigation

Modify ↓

#19972 closed enhancement (fixed)

Resolve libs using Ivy

Reported by:	simon04	Owned by:	simon04
Priority:	normal	Milestone:
Component:	Plugin pbf	Version:
Keywords:		Cc:	Don-vip

Description

I've finally managed to get OSM-binary deployed to Maven Central, see https://github.com/openstreetmap/OSM-binary/pull/44

For building the pbf plugin, we can now resolve its libs using Ivy…

Attachments (4)

19985-eclipse.patch (1.9 KB ) - added by GerdP 5 years ago.: This patch allows me to compile again in Eclipse. Please improve, I really don't understand the details. This is just a copy+paste solution.
osmpbf-1.5.0-SNAPSHOT.jar (138.1 KB ) - added by simon04 5 years ago.
pbf-build.log (11.0 KB ) - added by GerdP 5 years ago.: compile fails, get the same problem with the jars for mkgmap
19972-with-mkgmap-jars.patch (6.2 KB ) - added by GerdP 5 years ago.

Download all attachments as: .zip

Change History (27)

comment:1 by simon04, 5 years ago

Resolution:	→ fixed
Status:	assigned → closed

In 35604/osm:

fix #19972 - pbf: resolve libs using Ivy

comment:2 by simon04, 5 years ago

In 35605/osm:

see #19972 - pbf: cleanup

comment:3 by GerdP, 5 years ago

Great! The new pbf.jar is much larger than before (1.956K instead of 1.174K). Do we need them all?
Edit: Seems the protobuf compiler is now included?

Last edited 5 years ago by GerdP (previous) (diff)

comment:4 by GerdP, 5 years ago

Resolution:	fixed
Status:	closed → reopened

comment:5 by GerdP, 5 years ago

Any thoughts? I think it is good to download the protobuf stuff with ivy, but I really don't like the effect on the jar file.

comment:6 by Don-vip, 5 years ago

Agreed, 1 Mb plugin was already big, 2 Mb is too much, we should filter out the unneeded stuff.

comment:7 by GerdP, 5 years ago

Risking to state the obvious:
For the mkgmap project we still use a osmpbf-1.3.3.jar and a protobuf-java-2.5.0.jar. Both together have < 700Kb, so the plugin should not be much more than this even with the latest versions of the above.

follow-up: 9 comment:8 by simon04, 5 years ago

The old protobuf-java-2.5.0.jar was what is now published as protobuf-javalite (see https://github.com/protocolbuffers/protobuf/blob/master/java/lite.md and https://github.com/protocolbuffers/protobuf/releases/tag/v3.0.0)

Trying to build the pbf plugin with a snapshot version of osmbinary using protobuf-javalite (https://github.com/simon04/OSM-binary/commit/b8ebac4cddb2084c97546c61a86140a4a4082484) breaks all unit tests

org.openstreetmap.josm.io.IllegalDataException: com.google.protobuf.InvalidProtocolBufferException: While parsing a protocol message, the input ended unexpectedly in the middle of a field.  This could mean either that the input has been truncated or that an embedded message misreported its own length.

	at org.openstreetmap.josm.plugins.pbf.io.PbfReader.doParseDataSet(PbfReader.java:362)
	at org.openstreetmap.josm.plugins.pbf.io.PbfReader.parseDataSet(PbfReader.java:331)
	at org.openstreetmap.josm.plugins.pbf.io.PbfImporter.parseDataSet(PbfImporter.java:30)
	at org.openstreetmap.josm.plugins.pbf.io.PbfImporter.parseDataSet(PbfImporter.java:35)
	at org.openstreetmap.josm.plugins.pbf.io.PbfImporterTest.doTestMonaco(PbfImporterTest.java:41)
	at org.openstreetmap.josm.plugins.pbf.io.PbfImporterTest.testTicket10132(PbfImporterTest.java:67)
	at org.openstreetmap.josm.testutils.JOSMTestRules$TimeoutThread.run(JOSMTestRules.java:777)
Caused by: com.google.protobuf.InvalidProtocolBufferException: While parsing a protocol message, the input ended unexpectedly in the middle of a field.  This could mean either that the input has been truncated or that an embedded message misreported its own length.
	at com.google.protobuf.InvalidProtocolBufferException.truncatedMessage(InvalidProtocolBufferException.java:84)
	at com.google.protobuf.GeneratedMessageLite.parsePartialFrom(GeneratedMessageLite.java:1572)
	at com.google.protobuf.GeneratedMessageLite.parseFrom(GeneratedMessageLite.java:1659)
	at crosby.binary.Fileformat$BlobHeader.parseFrom(Fileformat.java:1021)
	at crosby.binary.file.FileBlockHead.readHead(FileBlockHead.java:57)
	at crosby.binary.file.FileBlock.process(FileBlock.java:130)
	at crosby.binary.file.BlockInputStream.process(BlockInputStream.java:34)
	at org.openstreetmap.josm.plugins.pbf.io.PbfReader.parse(PbfReader.java:370)
	at org.openstreetmap.josm.plugins.pbf.io.PbfReader.doParseDataSet(PbfReader.java:347)
	... 6 more


org.openstreetmap.josm.io.IllegalDataException: com.google.protobuf.InvalidProtocolBufferException: While parsing a protocol message, the input ended unexpectedly in the middle of a field.  This could mean either that the input has been truncated or that an embedded message misreported its own length.

	at org.openstreetmap.josm.plugins.pbf.io.PbfReader.doParseDataSet(PbfReader.java:362)
	at org.openstreetmap.josm.plugins.pbf.io.PbfReader.parseDataSet(PbfReader.java:331)
	at org.openstreetmap.josm.plugins.pbf.io.PbfImporter.parseDataSet(PbfImporter.java:30)
	at org.openstreetmap.josm.plugins.pbf.io.PbfImporter.parseDataSet(PbfImporter.java:35)
	at org.openstreetmap.josm.plugins.pbf.io.PbfImporterTest.testTicket12567(PbfImporterTest.java:76)
	at org.openstreetmap.josm.testutils.JOSMTestRules$TimeoutThread.run(JOSMTestRules.java:777)
Caused by: com.google.protobuf.InvalidProtocolBufferException: While parsing a protocol message, the input ended unexpectedly in the middle of a field.  This could mean either that the input has been truncated or that an embedded message misreported its own length.
	at com.google.protobuf.InvalidProtocolBufferException.truncatedMessage(InvalidProtocolBufferException.java:84)
	at com.google.protobuf.GeneratedMessageLite.parsePartialFrom(GeneratedMessageLite.java:1572)
	at com.google.protobuf.GeneratedMessageLite.parseFrom(GeneratedMessageLite.java:1659)
	at crosby.binary.Fileformat$BlobHeader.parseFrom(Fileformat.java:1021)
	at crosby.binary.file.FileBlockHead.readHead(FileBlockHead.java:57)
	at crosby.binary.file.FileBlock.process(FileBlock.java:130)
	at crosby.binary.file.BlockInputStream.process(BlockInputStream.java:34)
	at org.openstreetmap.josm.plugins.pbf.io.PbfReader.parse(PbfReader.java:370)
	at org.openstreetmap.josm.plugins.pbf.io.PbfReader.doParseDataSet(PbfReader.java:347)
	... 5 more


org.openstreetmap.josm.io.IllegalDataException: com.google.protobuf.InvalidProtocolBufferException: While parsing a protocol message, the input ended unexpectedly in the middle of a field.  This could mean either that the input has been truncated or that an embedded message misreported its own length.

	at org.openstreetmap.josm.plugins.pbf.io.PbfReader.doParseDataSet(PbfReader.java:362)
	at org.openstreetmap.josm.plugins.pbf.io.PbfReader.parseDataSet(PbfReader.java:331)
	at org.openstreetmap.josm.plugins.pbf.io.PbfImporter.parseDataSet(PbfImporter.java:30)
	at org.openstreetmap.josm.plugins.pbf.io.PbfImporter.parseDataSet(PbfImporter.java:35)
	at org.openstreetmap.josm.plugins.pbf.io.PbfImporterTest.testTicket14545(PbfImporterTest.java:89)
	at org.openstreetmap.josm.testutils.JOSMTestRules$TimeoutThread.run(JOSMTestRules.java:777)
Caused by: com.google.protobuf.InvalidProtocolBufferException: While parsing a protocol message, the input ended unexpectedly in the middle of a field.  This could mean either that the input has been truncated or that an embedded message misreported its own length.
	at com.google.protobuf.InvalidProtocolBufferException.truncatedMessage(InvalidProtocolBufferException.java:84)
	at com.google.protobuf.GeneratedMessageLite.parsePartialFrom(GeneratedMessageLite.java:1572)
	at com.google.protobuf.GeneratedMessageLite.parseFrom(GeneratedMessageLite.java:1659)
	at crosby.binary.Fileformat$BlobHeader.parseFrom(Fileformat.java:1021)
	at crosby.binary.file.FileBlockHead.readHead(FileBlockHead.java:57)
	at crosby.binary.file.FileBlock.process(FileBlock.java:130)
	at crosby.binary.file.BlockInputStream.process(BlockInputStream.java:34)
	at org.openstreetmap.josm.plugins.pbf.io.PbfReader.parse(PbfReader.java:370)
	at org.openstreetmap.josm.plugins.pbf.io.PbfReader.doParseDataSet(PbfReader.java:347)
	... 5 more


org.openstreetmap.josm.io.IllegalDataException: com.google.protobuf.InvalidProtocolBufferException: While parsing a protocol message, the input ended unexpectedly in the middle of a field.  This could mean either that the input has been truncated or that an embedded message misreported its own length.

	at org.openstreetmap.josm.plugins.pbf.io.PbfReader.doParseDataSet(PbfReader.java:362)
	at org.openstreetmap.josm.plugins.pbf.io.PbfReader.parseDataSet(PbfReader.java:331)
	at org.openstreetmap.josm.plugins.pbf.io.PbfImporter.parseDataSet(PbfImporter.java:30)
	at org.openstreetmap.josm.plugins.pbf.io.PbfImporter.parseDataSet(PbfImporter.java:35)
	at org.openstreetmap.josm.plugins.pbf.io.PbfImporterTest.doTestMonaco(PbfImporterTest.java:41)
	at org.openstreetmap.josm.plugins.pbf.io.PbfImporterTest.testParseDataSet(PbfImporterTest.java:58)
	at org.openstreetmap.josm.testutils.JOSMTestRules$TimeoutThread.run(JOSMTestRules.java:777)
Caused by: com.google.protobuf.InvalidProtocolBufferException: While parsing a protocol message, the input ended unexpectedly in the middle of a field.  This could mean either that the input has been truncated or that an embedded message misreported its own length.
	at com.google.protobuf.InvalidProtocolBufferException.truncatedMessage(InvalidProtocolBufferException.java:84)
	at com.google.protobuf.GeneratedMessageLite.parsePartialFrom(GeneratedMessageLite.java:1572)
	at com.google.protobuf.GeneratedMessageLite.parseFrom(GeneratedMessageLite.java:1659)
	at crosby.binary.Fileformat$BlobHeader.parseFrom(Fileformat.java:1021)
	at crosby.binary.file.FileBlockHead.readHead(FileBlockHead.java:57)
	at crosby.binary.file.FileBlock.process(FileBlock.java:130)
	at crosby.binary.file.BlockInputStream.process(BlockInputStream.java:34)
	at org.openstreetmap.josm.plugins.pbf.io.PbfReader.parse(PbfReader.java:370)
	at org.openstreetmap.josm.plugins.pbf.io.PbfReader.doParseDataSet(PbfReader.java:347)
	... 6 more


java.lang.ArrayIndexOutOfBoundsException: Index 5 out of bounds for length 5

	at com.google.protobuf.MessageSchema.newSchemaForRawMessageInfo(MessageSchema.java:507)
	at com.google.protobuf.MessageSchema.newSchema(MessageSchema.java:227)
	at com.google.protobuf.ManifestSchemaFactory.newSchema(ManifestSchemaFactory.java:77)
	at com.google.protobuf.ManifestSchemaFactory.createSchema(ManifestSchemaFactory.java:71)
	at com.google.protobuf.Protobuf.schemaFor(Protobuf.java:93)
	at com.google.protobuf.Protobuf.schemaFor(Protobuf.java:107)
	at com.google.protobuf.GeneratedMessageLite.makeImmutable(GeneratedMessageLite.java:175)
	at com.google.protobuf.GeneratedMessageLite$Builder.buildPartial(GeneratedMessageLite.java:395)
	at com.google.protobuf.GeneratedMessageLite$Builder.build(GeneratedMessageLite.java:403)
	at crosby.binary.Osmformat$HeaderBlock$Builder.setBbox(Osmformat.java:930)
	at org.openstreetmap.josm.plugins.pbf.io.PbfWriter$PbfSerializer.processBounds(PbfWriter.java:446)
	at org.openstreetmap.josm.plugins.pbf.io.PbfWriter$PbfSerializer$Processor.processSources(PbfWriter.java:358)
	at org.openstreetmap.josm.plugins.pbf.io.PbfWriter$PbfSerializer.process(PbfWriter.java:484)
	at org.openstreetmap.josm.plugins.pbf.io.PbfWriter.writeData(PbfWriter.java:515)
	at org.openstreetmap.josm.plugins.pbf.io.PbfWriter.writeLayer(PbfWriter.java:507)
	at org.openstreetmap.josm.plugins.pbf.io.PbfExporter.doSave(PbfExporter.java:34)
	at org.openstreetmap.josm.plugins.pbf.io.PbfExporterTest.testTicket11169(PbfExporterTest.java:39)
	at org.openstreetmap.josm.testutils.JOSMTestRules$TimeoutThread.run(JOSMTestRules.java:777)

in reply to: 8 comment:9 by GerdP, 5 years ago

Replying to simon04:

The old protobuf-java-2.5.0.jar was what is now published as protobuf-javalite (see https://github.com/protocolbuffers/protobuf/blob/master/java/lite.md and https://github.com/protocolbuffers/protobuf/releases/tag/v3.0.0)

Trying to build the pbf plugin with a snapshot version of osmbinary using protobuf-javalite (https://github.com/simon04/OSM-binary/commit/b8ebac4cddb2084c97546c61a86140a4a4082484) breaks all unit tests

How can I reproduce your results?

comment:10 by simon04, 5 years ago

Clone this branch: https://github.com/simon04/OSM-binary/compare/java-1.5.0-lite
Build using mvn clean package – the JAR files are placed in target/
Apply the following patch for the pbf plugin, adapt /your/path/to/, and run ant clean test

plugins/pbf/ivy.xml

diff --git a/plugins/pbf/ivy.xml b/plugins/pbf/ivy.xml
index 58c63dec7..720944beb 100644

-              a
 <ivy-module version="2.0">
   <info organisation="org.openstreetmap.josm.plugins" module="pbf"/>
   <dependencies>
+    <dependency conf="default->default" org="org.openstreetmap.pbf" name="osmpbf" rev="1.4.0"/>
+    <dependency conf="default->default" org="com.google.protobuf" name="protobuf-javalite" rev="3.13.0"/>
+    <dependency conf="default->default" org="org.openstreetmap.pbf" name="osmpbf" rev="1.5.0-SNAPSHOT"/>
   </dependencies>
 </ivy-module>

plugins/pbf/ivy_settings.xml

diff --git a/plugins/pbf/ivy_settings.xml b/plugins/pbf/ivy_settings.xml
index 847930cd7..3dd6ae8ea 100644

-              a
 <?xml version="1.0" encoding="utf-8"?>
 <ivysettings>
   <settings defaultResolver="josm-nexus"/>
+  <settings defaultResolver="chain-resolver"/>
   <resolvers>
+    <ibiblio name="josm-nexus" m2compatible="true" root="https://josm.openstreetmap.de/nexus/content/repositories/public/" />
+    <chain name="chain-resolver">
+      <filesystem name="osmpbf-snapshot">
+        <ivy
+                pattern= "/your/path/to/OSM-binary/target/[artifact]-[revision].[ext]" />
+        <artifact
+                pattern= "/your/path/to/OSM-binary/target/[artifact]-[revision].[ext]" />
+      </filesystem>
+      <ibiblio name="josm-nexus" m2compatible="true" root="https://josm.openstreetmap.de/nexus/content/repositories/public/" />
+    </chain>
   </resolvers>
 </ivysettings>

comment:11 by GerdP, 5 years ago

Hmm, mit git client seems to old, and I don't have maven installed on my Windows machine. Lost in toolchain...
I'll try the opposite approach: Use the jar files for the mkgmap project.

comment:12 by GerdP, 5 years ago

Next problem is that I don't know how to tell Eclipse to use the new ivy configuration for the pbf plugin. The files .project and .classpath need to be updated.

by GerdP, 5 years ago

Attachment:	19985-eclipse.patch added

This patch allows me to compile again in Eclipse. Please improve, I really don't understand the details. This is just a copy+paste solution.

comment:13 by GerdP, 5 years ago

No, sorry, it doesn't compile. I give up here and wait for the experts to fix this.

by simon04, 5 years ago

Attachment:	osmpbf-1.5.0-SNAPSHOT.jar added

by GerdP, 5 years ago

Attachment:	pbf-build.log added

compile fails, get the same problem with the jars for mkgmap

comment:14 by GerdP, 5 years ago

OK, getting further with this patch:

src/org/openstreetmap/josm/plugins/pbf/io/PbfReader.java

C:\josm\plugins\pbf>svn diff src

             if (header.hasBbox()) {
                 HeaderBBox bbox = header.getBbox();
+                if (bbox != null) {
+                    double minlat = parseRawDegrees(bbox.getBottom());
+                    double minlon = parseRawDegrees(bbox.getLeft());
+                    double maxlat = parseRawDegrees(bbox.getTop());
+                    double maxlon = parseRawDegrees(bbox.getRight());
+                    Bounds b = new Bounds(minlat, minlon, maxlat, maxlon);
+                    if (!b.isCollapsed() && areCoordinatesValid(minlat, minlon, maxlat, maxlon)) {
+                        ds.addDataSource(new DataSource(b, header.getSource()));
+                    } else {
+                        Logging.error("Invalid Bounds: "+b);
+                    }
+                double minlat = parseRawDegrees(bbox.getBottom());
+                double minlon = parseRawDegrees(bbox.getLeft());
+                double maxlat = parseRawDegrees(bbox.getTop());
+                double maxlon = parseRawDegrees(bbox.getRight());
+                Bounds b = new Bounds(minlat, minlon, maxlat, maxlon);
+                if (!b.isCollapsed() && areCoordinatesValid(minlat, minlon, maxlat, maxlon)) {
+                       ds.addDataSource(new DataSource(b, header.getSource()));
+                } else {
+                       Logging.error("Invalid Bounds: "+b);
+                }
+            }
+        }

by GerdP, 5 years ago

Attachment:	19972-with-mkgmap-jars.patch added

comment:15 by GerdP, 5 years ago

This patch uses the jars from the mkgmap project. The unit tests work and a quick check in JOSM also looks good. pbf.jar is 718K. Hope it helps somehow.

comment:16 by GerdP, 5 years ago

The error messages for osmpbf-1.5.0-SNAPSHOT.jar look like a problem with the protobuf compiler. I wonder why the generated *.java sources in your branch are eqaul to those in https://github.com/openstreetmap/OSM-binary
I would have expected that they are different.

follow-up: 18 comment:17 by GerdP, 5 years ago

No, sorry, looked at the wrong files. I can reproduce the error messages with your jar here.
Did you try without the line
<outputOptions>lite</outputOptions>
in pom.xml?

in reply to: 17 comment:18 by simon04, 5 years ago

Replying to GerdP:

Did you try without the line
<outputOptions>lite</outputOptions>
in pom.xml?

You cannot use <artifactId>protobuf-javalite</artifactId> without compiling with <outputOptions>lite</outputOptions> –

$ mvn clean package
…
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile (default-compile) on project osmpbf: Compilation failure: Compilation failure: 
[ERROR] /home/simon/src/OSM-binary/target/generated-sources/crosby/binary/Fileformat.java:[120,26] cannot find symbol
[ERROR]   symbol:   class GeneratedMessageV3
[ERROR]   location: package com.google.protobuf
[ERROR] /home/simon/src/OSM-binary/target/generated-sources/crosby/binary/Fileformat.java:[19,26] cannot find symbol
[ERROR]   symbol:   class MessageOrBuilder
[ERROR]   location: package com.google.protobuf
[ERROR] /home/simon/src/OSM-binary/target/generated-sources/crosby/binary/Fileformat.java:[577,47] package com.google.protobuf.GeneratedMessageV3 does not exist

comment:19 by GerdP, 5 years ago

OK. I did not follow the development of protobufs for a while. IIGTR the 2.5 version is still maintained. Why not use that?

comment:20 by GerdP, 5 years ago

Ping

comment:21 by simon04, 5 years ago

Resolution:	→ fixed
Status:	reopened → closed

In 35631/osm:

fix #19972 - pbf: downgrade to protobuf-java 2.5.0, downgrade to osmpbf 1.3.3, resolve osmpbf from mkgmap.org.uk

comment:22 by GerdP, 5 years ago

In 35633/osm:

see #19972: Resolve libs using Ivy

update eclipse files (hope I got it right)

comment:23 by GerdP, 5 years ago

In 35635/osm:

see #19972: Resolve libs using Ivy

some cleanup

Modify Ticket

Change Properties

Summary:
Type:		Priority:
Milestone:		Component:
Version:		Keywords:
Cc:	Set your email in Preferences

Action

leave as closed The owner will remain simon04.

change resolution as The resolution will be set.

reopen The resolution will be deleted. Next status will be 'reopened'.

Add Comment

Your email or username:

E-mail address and name can be saved in the Preferences .

You may use WikiFormatting here.

Attachments ↑ Description ↑

Note: See TracTickets for help on using tickets.

Download in other formats:

Context Navigation

#19972 closed enhancement (fixed)

Resolve libs using Ivy

Description

Attachments (4)

Change History (27)

comment:1 by simon04, 5 years ago

comment:2 by simon04, 5 years ago

comment:3 by GerdP, 5 years ago

comment:4 by GerdP, 5 years ago

comment:5 by GerdP, 5 years ago

comment:6 by Don-vip, 5 years ago

comment:7 by GerdP, 5 years ago

follow-up: 9 comment:8 by simon04, 5 years ago

in reply to: 8 comment:9 by GerdP, 5 years ago

comment:10 by simon04, 5 years ago

plugins/pbf/ivy.xml

plugins/pbf/ivy_settings.xml

comment:11 by GerdP, 5 years ago

comment:12 by GerdP, 5 years ago

by GerdP, 5 years ago

comment:13 by GerdP, 5 years ago

by simon04, 5 years ago

by GerdP, 5 years ago

comment:14 by GerdP, 5 years ago

src/org/openstreetmap/josm/plugins/pbf/io/PbfReader.java

by GerdP, 5 years ago

comment:15 by GerdP, 5 years ago

comment:16 by GerdP, 5 years ago

follow-up: 18 comment:17 by GerdP, 5 years ago

in reply to: 17 comment:18 by simon04, 5 years ago

comment:19 by GerdP, 5 years ago

comment:20 by GerdP, 5 years ago

comment:21 by simon04, 5 years ago

comment:22 by GerdP, 5 years ago

comment:23 by GerdP, 5 years ago

Modify Ticket

Add Comment

by anonymous

Download in other formats: