Modify

Opened 5 years ago

Closed 3 years ago

#10132 closed defect (fixed)

IndexOutOfBoundsException while reading PBF

Reported by: naoliv Owned by: Don-vip
Priority: normal Milestone:
Component: Plugin pbf Version:
Keywords: github Cc:

Description

While trying to read http://naoliv.iq.unesp.br/osm/Brasil-SP.osm.pbf I've got this:

ERRO: org.openstreetmap.josm.io.IllegalDataException: java.lang.IndexOutOfBoundsException: Index: 0. Causa: java.lang.IndexOutOfBoundsException: Index: 0
org.openstreetmap.josm.io.IllegalDataException: java.lang.IndexOutOfBoundsException: Index: 0
	at org.openstreetmap.josm.plugins.pbf.io.PbfReader.parseDataSet(PbfReader.java:267)
	at org.openstreetmap.josm.plugins.pbf.io.PbfImporter.parseDataSet(PbfImporter.java:28)
	at org.openstreetmap.josm.io.OsmImporter.loadLayer(OsmImporter.java:110)
	at org.openstreetmap.josm.io.OsmImporter.importData(OsmImporter.java:88)
	at org.openstreetmap.josm.io.OsmImporter.importData(OsmImporter.java:65)
	at org.openstreetmap.josm.io.FileImporter.importDataHandleExceptions(FileImporter.java:64)
	at org.openstreetmap.josm.actions.OpenFileAction$OpenFileTask.importData(OpenFileAction.java:332)
	at org.openstreetmap.josm.actions.OpenFileAction$OpenFileTask.realRun(OpenFileAction.java:277)
	at org.openstreetmap.josm.gui.PleaseWaitRunnable.doRealRun(PleaseWaitRunnable.java:82)
	at org.openstreetmap.josm.gui.PleaseWaitRunnable.run(PleaseWaitRunnable.java:150)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.IndexOutOfBoundsException: Index: 0
	at java.util.Collections$EmptyList.get(Collections.java:3212)
	at crosby.binary.Osmformat$DenseInfo.getChangeset(Osmformat.java:3803)
	at org.openstreetmap.josm.plugins.pbf.io.PbfReader$PbfParser.parseDense(PbfReader.java:94)
	at crosby.binary.BinaryParser.parse(BinaryParser.java:124)
	at crosby.binary.BinaryParser.handleBlock(BinaryParser.java:68)
	at crosby.binary.file.FileBlock.process(FileBlock.java:135)
	at crosby.binary.file.BlockInputStream.process(BlockInputStream.java:34)
	at org.openstreetmap.josm.plugins.pbf.io.PbfReader.parse(PbfReader.java:274)
	at org.openstreetmap.josm.plugins.pbf.io.PbfReader.parseDataSet(PbfReader.java:257)
	... 14 more

JOSM:

Repository Root: http://josm.openstreetmap.de/svn
Build-Date: 2014-06-09 01:34:44
Last Changed Author: akks
Revision: 7224
Repository UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
URL: http://josm.openstreetmap.de/svn/trunk
Last Changed Date: 2014-06-08 22:01:41 +0200 (Sun, 08 Jun 2014)
Last Changed Rev: 7224

Identification: JOSM/1.5 (7224 pt_BR) Linux Debian GNU/Linux testing (jessie)
Memory Usage: 646 MB / 6110 MB (223 MB allocated, but free)
Java version: 1.7.0_55, Oracle Corporation, OpenJDK 64-Bit Server VM
Java package: openjdk-7-jre:amd64-7u55-2.4.7-2
VM arguments: [-Djava.net.useSystemProxies=true, -Dawt.useSystemAAFontSettings=on, -Xms512M, -Xmx6g, -XX:+UseConcMarkSweepGC, -XX:+UseCompressedOops]

Plugin: Create_grid_of_ways (30416)
Plugin: OpeningHoursEditor (30416)
Plugin: PicLayer (30436)
Plugin: SimplifyArea (30416)
Plugin: buildings_tools (30485)
Plugin: contourmerge (1010)
Plugin: editgpx (30416)
Plugin: geotools (30416)
Plugin: importvec (30416)
Plugin: jts (30416)
Plugin: log4j (30416)
Plugin: measurement (30416)
Plugin: merge-overlap (30416)
Plugin: opendata (30436)
Plugin: pbf (30416)
Plugin: pdfimport (30416)
Plugin: poly (30416)
Plugin: reverter (30436)
Plugin: scripting (30604)
Plugin: todo (29154)
Plugin: turnrestrictions (30454)
Plugin: undelete (30416)
Plugin: utilsplugin2 (30460)

Attachments (1)

improve_pbf_plugin.patch (10.3 KB) - added by GerdP 3 years ago.

Download all attachments as: .zip

Change History (12)

comment:1 Changed 5 years ago by Don-vip

Owner: changed from Don-vip to naoliv
Status: newneedinfo

403 Forbidden :)

comment:2 Changed 5 years ago by naoliv

Sorry. Permission is now fixed.

comment:3 Changed 5 years ago by Don-vip

Owner: changed from naoliv to Don-vip
Status: needinfoassigned

comment:4 Changed 5 years ago by Don-vip

how did you build this file?

comment:5 Changed 5 years ago by naoliv

With osmconvert brazil.osm.pbf -B=SP.poly --complex-ways --complete-ways --drop-author -o=Brasil-SP.osm.pbf

Where:
osconvert = version 0.7T
brazil.osm.pbf = http://download.geofabrik.de/south-america/brazil-latest.osm.pbf
SP.poly = http://naoliv.iq.unesp.br/osm/poly/SP.poly

comment:6 Changed 5 years ago by naoliv

I was taking a look at http://wiki.openstreetmap.org/wiki/Osmconvert#Exclude_Information_or_Contents_from_the_Output_File:

"Usually you will not encounter any problems when deleting the author information from .osm or .o5m files, however it is not encouraged to do this with .pbf files because most programs will not cope with this change of format."

So it seems that I am creating a somehow invalid file.
Can I convert my bug report to "Please, give an user-friendly message saying that no author info was found in the PBF file"? :-)

comment:7 Changed 5 years ago by Don-vip

Keywords: github added

The problem is that if fails inside my PBF library and not the plugin itself. The ticket should be forwarded to https://github.com/scrosby/OSM-binary/issues :)

comment:8 Changed 5 years ago by Don-vip

Resolution: othersoftware
Status: assignedclosed

comment:9 Changed 4 years ago by GerdP

Resolution: othersoftware
Status: closedreopened

I think the problem can also be fixed in the plugin code.
It contains code like this:
Node node = new Node(nodeId+=nodes.getId(i), nodes.getDenseinfo().getVersion(i));

The getDenseInfo() method should only be used after checking
that nodes.hasDenseInfo() returns true.
A similar problem may be here:

final Info info = w.getInfo();

final Way way = new Way(w.getId(), info.getVersion());

The code should first check that w.hasInfo() returns true.

Changed 3 years ago by GerdP

Attachment: improve_pbf_plugin.patch added

comment:10 Changed 3 years ago by GerdP

I've attached a patch which implements the suggested tests.
This allows to import pbf files which were created by osmconvert with
--drop-version or --drop-author and also files produced by tile splitter (mkgmap)

comment:11 Changed 3 years ago by Don-vip

Resolution: fixed
Status: reopenedclosed

Thanks! Fixed in [o32859:32860].

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The owner will remain Don-vip.
as The resolution will be set.
The resolution will be deleted.

Add Comment


E-mail address and name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.