Package org.openstreetmap.josm.io

Class OsmPbfReader

java.lang.Object

org.openstreetmap.josm.io.AbstractReader

org.openstreetmap.josm.io.OsmPbfReader

public final class OsmPbfReader
extends AbstractReader

Read OSM data from an OSM PBF file

Since:

18695

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
private static class	OsmPbfReader.BoundedInputStream	This could be replaced by BoundedInputStream from Apache Commons IO.
private static class	OsmPbfReader.PrimitiveBlockRecord	A record class for passing PrimitiveBlock information to the PrimitiveGroup parser

Nested classes/interfaces inherited from class org.openstreetmap.josm.io.AbstractReader

AbstractReader.BinaryParserWorker, AbstractReader.CommonReader, AbstractReader.NodeReader, AbstractReader.ParserWorker, AbstractReader.RelationReader, AbstractReader.WayReader

Field Summary

Fields

Modifier and Type	Field	Description
private static long[]	EMPTY_LONG
private static int	MAX_BLOB_SIZE	The maximum Blob size.
private static int	MAX_BLOBHEADER_SIZE	The maximum BlobHeader size.
private static double	NANO_DEGREES	Nano degrees

Fields inherited from class org.openstreetmap.josm.io.AbstractReader

cancel, ds, externalIdMap, relations, uploadChangeset, ways

Constructor Summary

Constructors

Modifier	Constructor	Description
private	OsmPbfReader()

Method Summary

Modifier and Type	Method	Description
private static void	addTags(Tagged primitive, java.util.List<java.lang.String> keys, java.util.List<java.lang.String> values)	Add a set of tags to a primitive
private static LatLon	calculateLatLon(OsmPbfReader.PrimitiveBlockRecord primitiveBlockRecord, long lat, long lon)	Calculate the actual lat lon
private static void	checkRequiredFeatures(HeaderBlock headerBlock)	Ensure that we support all the required features in the PBF
private static long[]	decodePackedSInt64(long[] numbers)	Convert an array of numbers to an array of longs, decoded from uint (zig zag decoded)
protected DataSet	doParseDataSet(java.io.InputStream source, ProgressMonitor progressMonitor)
private static long[]	joinArrays(long[] array1, long[] array2)	Join two different arrays
private void	parse(java.io.InputStream source)
private static BBox	parseBBox(java.io.ByteArrayOutputStream baos, ProtobufRecord current)	This parses a bbox from a record (HeaderBBox message)
private static Blob	parseBlob(BlobHeader header, OsmPbfReader.BoundedInputStream cis, ProtobufParser parser, java.io.ByteArrayOutputStream baos)	Parse a blob from the PBF file
private static BlobHeader	parseBlobHeader(OsmPbfReader.BoundedInputStream cis, java.io.ByteArrayOutputStream baos, ProtobufParser parser)	Parse a blob header
private void	parseDataBlock(java.io.ByteArrayOutputStream baos, HeaderBlock headerBlock, Blob blob)	Parse a data blob (should be "OSMData")
static DataSet	parseDataSet(java.io.InputStream source, ProgressMonitor progressMonitor)	Parse the given input source and return the dataset.
private static Info[]	parseDenseInfo(java.io.ByteArrayOutputStream baos, byte[] bytes)	Parse dense info
private void	parseDenseNodes(java.io.ByteArrayOutputStream baos, byte[] bytes, OsmPbfReader.PrimitiveBlockRecord primitiveBlockRecord)	Parse dense nodes from a record
private static HeaderBlock	parseHeaderBlock(Blob blob, java.io.ByteArrayOutputStream baos)	Parse a header block.
private static Info	parseInfo(java.io.ByteArrayOutputStream baos, byte[] bytes)	Parse info for an object
private void	parseNode(java.io.ByteArrayOutputStream baos, byte[] bytes, OsmPbfReader.PrimitiveBlockRecord primitiveBlockRecord)	Parse a singular node
private void	parsePrimitiveGroup(java.io.ByteArrayOutputStream baos, byte[] bytes, OsmPbfReader.PrimitiveBlockRecord primitiveBlockRecord)	Parse a PrimitiveGroup.
private void	parseRelation(java.io.ByteArrayOutputStream baos, byte[] bytes, OsmPbfReader.PrimitiveBlockRecord primitiveBlockRecord)	Parse a relation from a PBF
private static java.lang.String[]	parseStringTable(java.io.ByteArrayOutputStream baos, byte[] bytes)	Parse the string table
private void	parseWay(java.io.ByteArrayOutputStream baos, byte[] bytes, OsmPbfReader.PrimitiveBlockRecord primitiveBlockRecord)	Parse a way from the PBF
private static void	setOsmPrimitiveData(OsmPbfReader.PrimitiveBlockRecord primitiveBlockRecord, PrimitiveData primitive, Info info)	Set the primitive data for an object

Methods inherited from class org.openstreetmap.josm.io.AbstractReader

buildPrimitive, callPostProcessors, deregisterPostprocessor, doParseDataSet, doParseDataSet, getDataSet, getLong, parseAction, parseBounds, parseChangeset, parseChangeset, parseDownloadPolicy, parseId, parseLocked, parseNode, parseNode, parseRelation, parseRelationMember, parseRelationMember, parseTag, parseTimestamp, parseUploadPolicy, parseUser, parseUser, parseVersion, parseVersion, parseVersion, parseVisible, parseWay, prepareDataSet, processChangesetAfterParsing, processNodesAfterParsing, processRelationsAfterParsing, processWaysAfterParsing, registerPostprocessor

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

EMPTY_LONG

private static final long[] EMPTY_LONG

NANO_DEGREES

private static final double NANO_DEGREES

Nano degrees

See Also:

Constant Field Values

MAX_BLOBHEADER_SIZE

private static final int MAX_BLOBHEADER_SIZE

The maximum BlobHeader size. BlobHeaders should (but not must) be less than half this

See Also:

Constant Field Values

MAX_BLOB_SIZE

private static final int MAX_BLOB_SIZE

The maximum Blob size. Blobs should (but not must) be less than half this

See Also:

Constant Field Values

Constructor Detail

OsmPbfReader

private OsmPbfReader()

Method Detail

parseDataSet

public static DataSet parseDataSet(java.io.InputStream source,
                                   ProgressMonitor progressMonitor)
                            throws IllegalDataException

Parse the given input source and return the dataset.

Parameters:

source - the source input stream. Must not be null.

progressMonitor - the progress monitor. If null, NullProgressMonitor.INSTANCE is assumed

Returns:

the dataset with the parsed data

Throws:

IllegalDataException - if an error was found while parsing the data from the source

java.lang.IllegalArgumentException - if source is null

doParseDataSet

protected DataSet doParseDataSet(java.io.InputStream source,
                                 ProgressMonitor progressMonitor)
                          throws IllegalDataException

Specified by:

doParseDataSet in class AbstractReader

Throws:

IllegalDataException

parse

private void parse(java.io.InputStream source)
            throws IllegalDataException,
                   java.io.IOException

Throws:

IllegalDataException

java.io.IOException

parseBlobHeader

@Nonnull
private static BlobHeader parseBlobHeader(OsmPbfReader.BoundedInputStream cis,
                                          java.io.ByteArrayOutputStream baos,
                                          ProtobufParser parser)
                                   throws java.io.IOException,
                                          IllegalDataException

Parse a blob header

Parameters:

cis - A counting stream to ensure we don't read too much data

baos - A reusable stream

parser - The parser to read from

Returns:

The BlobHeader message

Throws:

java.io.IOException - if one of the streams has an issue

IllegalDataException - If the OSM PBF is (probably) corrupted

parseBlob

@Nonnull
private static Blob parseBlob(BlobHeader header,
                              OsmPbfReader.BoundedInputStream cis,
                              ProtobufParser parser,
                              java.io.ByteArrayOutputStream baos)
                       throws java.io.IOException

Parse a blob from the PBF file

Parameters:

header - The header with the blob information (most critically, the length of the blob)

cis - Used to ensure we don't read too much data

parser - The parser to read records from

baos - The reusable output stream

Returns:

The blob to use elsewhere

Throws:

java.io.IOException - If one of the streams has an issue

parseHeaderBlock

@Nonnull
private static HeaderBlock parseHeaderBlock(Blob blob,
                                            java.io.ByteArrayOutputStream baos)
                                     throws java.io.IOException

Parse a header block. This assumes that the parser has hit a string with the text "OSMHeader".

Parameters:

blob - The blob with the header block data

baos - The reusable output stream to use

Returns:

The parsed HeaderBlock

Throws:

java.io.IOException - if one of the InputStreams has a problem

checkRequiredFeatures

private static void checkRequiredFeatures(HeaderBlock headerBlock)
                                   throws IllegalDataException

Ensure that we support all the required features in the PBF

Parameters:

headerBlock - The HeaderBlock to check

Throws:

IllegalDataException - If there exists at least one feature that we do not support

parseDataBlock

private void parseDataBlock(java.io.ByteArrayOutputStream baos,
                            HeaderBlock headerBlock,
                            Blob blob)
                     throws java.io.IOException,
                            IllegalDataException

Parse a data blob (should be "OSMData")

Parameters:

baos - The reusable stream

headerBlock - The header block with data source information

blob - The blob to read OSM data from

Throws:

java.io.IOException - if we don't support the compression type

IllegalDataException - If an invalid OSM primitive was read

parseBBox

@Nullable
private static BBox parseBBox(java.io.ByteArrayOutputStream baos,
                              ProtobufRecord current)
                       throws java.io.IOException

This parses a bbox from a record (HeaderBBox message)

Parameters:

baos - The reusable ByteArrayOutputStream to avoid unnecessary allocations

current - The current record

Returns:

The immutable bbox, or null

Throws:

java.io.IOException - If something happens with the InputStreams (probably won't happen)

parseStringTable

@Nonnull
private static java.lang.String[] parseStringTable(java.io.ByteArrayOutputStream baos,
                                                   byte[] bytes)
                                            throws java.io.IOException

Parse the string table

Parameters:

baos - The reusable stream

bytes - The message bytes

Returns:

The parsed table (reminder: index 0 is empty, note that all strings are already interned by String.intern())

Throws:

java.io.IOException - if something happened while reading a ByteArrayInputStream

parsePrimitiveGroup

private void parsePrimitiveGroup(java.io.ByteArrayOutputStream baos,
                                 byte[] bytes,
                                 OsmPbfReader.PrimitiveBlockRecord primitiveBlockRecord)
                          throws IllegalDataException,
                                 java.io.IOException

Parse a PrimitiveGroup. Note: this parsing implementation doesn't check and make certain that all primitives in the group are the same type.

Parameters:

baos - The reusable stream

bytes - The bytes to decode

primitiveBlockRecord - The record to use for creating the primitives

Throws:

IllegalDataException - if one of the primitive records was invalid

java.io.IOException - if something happened while reading a ByteArrayInputStream

parseNode

private void parseNode(java.io.ByteArrayOutputStream baos,
                       byte[] bytes,
                       OsmPbfReader.PrimitiveBlockRecord primitiveBlockRecord)
                throws IllegalDataException,
                       java.io.IOException

Parse a singular node

Parameters:

baos - The reusable stream

bytes - The bytes to decode

primitiveBlockRecord - The record to use (mostly for tags and lat/lon calculations)

Throws:

IllegalDataException - if the PBF did not provide all the data necessary for node creation

java.io.IOException - if something happened while reading a ByteArrayInputStream

parseDenseNodes

private void parseDenseNodes(java.io.ByteArrayOutputStream baos,
                             byte[] bytes,
                             OsmPbfReader.PrimitiveBlockRecord primitiveBlockRecord)
                      throws IllegalDataException,
                             java.io.IOException

Parse dense nodes from a record

Parameters:

baos - The reusable output stream

bytes - The bytes for the dense node

primitiveBlockRecord - Used for data that is common between several different objects.

Throws:

IllegalDataException - if the nodes could not be parsed, or one of the nodes would be malformed

java.io.IOException - if something happened while reading a ByteArrayInputStream

parseWay

private void parseWay(java.io.ByteArrayOutputStream baos,
                      byte[] bytes,
                      OsmPbfReader.PrimitiveBlockRecord primitiveBlockRecord)
               throws IllegalDataException,
                      java.io.IOException

Parse a way from the PBF

Parameters:

baos - The reusable stream

bytes - The bytes for the way

primitiveBlockRecord - Used for common information, like tags

Throws:

IllegalDataException - if an invalid way could have been created

java.io.IOException - if something happened while reading a ByteArrayInputStream

parseRelation

private void parseRelation(java.io.ByteArrayOutputStream baos,
                           byte[] bytes,
                           OsmPbfReader.PrimitiveBlockRecord primitiveBlockRecord)
                    throws IllegalDataException,
                           java.io.IOException

Parse a relation from a PBF

Parameters:

baos - The reusable stream

bytes - The bytes to use

primitiveBlockRecord - Mostly used for tags

Throws:

IllegalDataException - if the PBF had a bad relation definition

java.io.IOException - if something happened while reading a ByteArrayInputStream

parseInfo

@Nonnull
private static Info parseInfo(java.io.ByteArrayOutputStream baos,
                              byte[] bytes)
                       throws java.io.IOException

Parse info for an object

Parameters:

baos - The reusable stream to use

bytes - The bytes to decode

Returns:

The info for an object

Throws:

java.io.IOException - if something happened while reading a ByteArrayInputStream

calculateLatLon

@Nonnull
private static LatLon calculateLatLon(OsmPbfReader.PrimitiveBlockRecord primitiveBlockRecord,
                                      long lat,
                                      long lon)

Calculate the actual lat lon

Parameters:

primitiveBlockRecord - The record with offset and granularity data

lat - The latitude from the PBF

lon - The longitude from the PBF

Returns:

The actual LatLon, accounting for PBF offset and granularity changes

addTags

private static void addTags(Tagged primitive,
                            java.util.List<java.lang.String> keys,
                            java.util.List<java.lang.String> values)

Add a set of tags to a primitive

Parameters:

primitive - The primitive to add tags to

keys - The keys (must match the size of the values)

values - The values (must match the size of the keys)

setOsmPrimitiveData

private static void setOsmPrimitiveData(OsmPbfReader.PrimitiveBlockRecord primitiveBlockRecord,
                                        PrimitiveData primitive,
                                        Info info)

Set the primitive data for an object

Parameters:

primitiveBlockRecord - The record with data for the current primitive (currently uses OsmPbfReader.PrimitiveBlockRecord.stringTable and OsmPbfReader.PrimitiveBlockRecord.dateGranularity).

primitive - The primitive to add the information to

info - The specific info for the primitive

decodePackedSInt64

@Nonnull
private static long[] decodePackedSInt64(long[] numbers)

Convert an array of numbers to an array of longs, decoded from uint (zig zag decoded)

Parameters:

numbers - The numbers to convert

Returns:

The long array (the same array that was passed in)

joinArrays

@Nonnull
private static long[] joinArrays(long[] array1,
                                 long[] array2)

Join two different arrays

Parameters:

array1 - The first array

array2 - The second array

Returns:

The joined arrays -- may return one of the original arrays, if the other is empty

parseDenseInfo

@Nonnull
private static Info[] parseDenseInfo(java.io.ByteArrayOutputStream baos,
                                     byte[] bytes)
                              throws IllegalDataException,
                                     java.io.IOException

Parse dense info

Parameters:

baos - The reusable stream

bytes - The bytes to decode

Returns:

The dense info array

Throws:

IllegalDataException - If the data has mismatched array lengths

java.io.IOException - if something happened while reading a ByteArrayInputStream