IndexOutOfBoundsException while reading PBF

[naoliv]

While trying to read ​http://naoliv.iq.unesp.br/osm/Brasil-SP.osm.pbf I've got this:

ERRO: org.openstreetmap.josm.io.IllegalDataException: java.lang.IndexOutOfBoundsException: Index: 0. Causa: java.lang.IndexOutOfBoundsException: Index: 0
org.openstreetmap.josm.io.IllegalDataException: java.lang.IndexOutOfBoundsException: Index: 0
	at org.openstreetmap.josm.plugins.pbf.io.PbfReader.parseDataSet(PbfReader.java:267)
	at org.openstreetmap.josm.plugins.pbf.io.PbfImporter.parseDataSet(PbfImporter.java:28)
	at org.openstreetmap.josm.io.OsmImporter.loadLayer(OsmImporter.java:110)
	at org.openstreetmap.josm.io.OsmImporter.importData(OsmImporter.java:88)
	at org.openstreetmap.josm.io.OsmImporter.importData(OsmImporter.java:65)
	at org.openstreetmap.josm.io.FileImporter.importDataHandleExceptions(FileImporter.java:64)
	at org.openstreetmap.josm.actions.OpenFileAction$OpenFileTask.importData(OpenFileAction.java:332)
	at org.openstreetmap.josm.actions.OpenFileAction$OpenFileTask.realRun(OpenFileAction.java:277)
	at org.openstreetmap.josm.gui.PleaseWaitRunnable.doRealRun(PleaseWaitRunnable.java:82)
	at org.openstreetmap.josm.gui.PleaseWaitRunnable.run(PleaseWaitRunnable.java:150)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.IndexOutOfBoundsException: Index: 0
	at java.util.Collections$EmptyList.get(Collections.java:3212)
	at crosby.binary.Osmformat$DenseInfo.getChangeset(Osmformat.java:3803)
	at org.openstreetmap.josm.plugins.pbf.io.PbfReader$PbfParser.parseDense(PbfReader.java:94)
	at crosby.binary.BinaryParser.parse(BinaryParser.java:124)
	at crosby.binary.BinaryParser.handleBlock(BinaryParser.java:68)
	at crosby.binary.file.FileBlock.process(FileBlock.java:135)
	at crosby.binary.file.BlockInputStream.process(BlockInputStream.java:34)
	at org.openstreetmap.josm.plugins.pbf.io.PbfReader.parse(PbfReader.java:274)
	at org.openstreetmap.josm.plugins.pbf.io.PbfReader.parseDataSet(PbfReader.java:257)
	... 14 more

JOSM:

Repository Root: http://josm.openstreetmap.de/svn
Build-Date: 2014-06-09 01:34:44
Last Changed Author: akks
Revision: 7224
Repository UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
URL: http://josm.openstreetmap.de/svn/trunk
Last Changed Date: 2014-06-08 22:01:41 +0200 (Sun, 08 Jun 2014)
Last Changed Rev: 7224

Identification: JOSM/1.5 (7224 pt_BR) Linux Debian GNU/Linux testing (jessie)
Memory Usage: 646 MB / 6110 MB (223 MB allocated, but free)
Java version: 1.7.0_55, Oracle Corporation, OpenJDK 64-Bit Server VM
Java package: openjdk-7-jre:amd64-7u55-2.4.7-2
VM arguments: [-Djava.net.useSystemProxies=true, -Dawt.useSystemAAFontSettings=on, -Xms512M, -Xmx6g, -XX:+UseConcMarkSweepGC, -XX:+UseCompressedOops]

Plugin: Create_grid_of_ways (30416)
Plugin: OpeningHoursEditor (30416)
Plugin: PicLayer (30436)
Plugin: SimplifyArea (30416)
Plugin: buildings_tools (30485)
Plugin: contourmerge (1010)
Plugin: editgpx (30416)
Plugin: geotools (30416)
Plugin: importvec (30416)
Plugin: jts (30416)
Plugin: log4j (30416)
Plugin: measurement (30416)
Plugin: merge-overlap (30416)
Plugin: opendata (30436)
Plugin: pbf (30416)
Plugin: pdfimport (30416)
Plugin: poly (30416)
Plugin: reverter (30436)
Plugin: scripting (30604)
Plugin: todo (29154)
Plugin: turnrestrictions (30454)
Plugin: undelete (30416)
Plugin: utilsplugin2 (30460)

[Don-vip]

403 Forbidden :)

[naoliv]

Sorry. Permission is now fixed.

[Don-vip]

how did you build this file?

[naoliv]

With osmconvert brazil.osm.pbf -B=SP.poly --complex-ways --complete-ways --drop-author -o=Brasil-SP.osm.pbf

Where:
osconvert = version 0.7T
brazil.osm.pbf = ​http://download.geofabrik.de/south-america/brazil-latest.osm.pbf
SP.poly = ​http://naoliv.iq.unesp.br/osm/poly/SP.poly

[naoliv]

I was taking a look at ​http://wiki.openstreetmap.org/wiki/Osmconvert#Exclude_Information_or_Contents_from_the_Output_File:

"Usually you will not encounter any problems when deleting the author information from .osm or .o5m files, however it is not encouraged to do this with .pbf files because most programs will not cope with this change of format."

So it seems that I am creating a somehow invalid file.
Can I convert my bug report to "Please, give an user-friendly message saying that no author info was found in the PBF file"? :-)

[Don-vip]

The problem is that if fails inside my PBF library and not the plugin itself. The ticket should be forwarded to ​https://github.com/scrosby/OSM-binary/issues :)

[GerdP]

I think the problem can also be fixed in the plugin code.
It contains code like this:
Node node = new Node(nodeId+=nodes.getId(i), nodes.getDenseinfo().getVersion(i));

The getDenseInfo() method should only be used after checking
that nodes.hasDenseInfo() returns true.
A similar problem may be here:

final Info info = w.getInfo();

final Way way = new Way(w.getId(), info.getVersion());

The code should first check that w.hasInfo() returns true.

[GerdP]

I've attached a patch which implements the suggested tests.
This allows to import pbf files which were created by osmconvert with
--drop-version or --drop-author and also files produced by tile splitter (mkgmap)

[Don-vip]

Thanks! Fixed in [o32859:32860].