Modify

Opened 16 months ago

Closed 13 months ago

Last modified 10 months ago

#22603 closed defect (fixed)

[PATCH] Add PBF reading support to JOSM

Reported by: taylor.smock Owned by: taylor.smock
Priority: normal Milestone: 23.03
Component: Core Version:
Keywords: pbf Cc:

Description (last modified by taylor.smock)

This is a followup of #6886.

Major differences:

  • It uses the JOSM Protobuf parsing code instead of Google's Protobuf parser
  • It is not based off of any of the code from https://github.com/openstreetmap/OSM-binary except for the proto files. I'm unsure what the licensing situation would require.
  • ~1500 added lines
  • Performance for PBF reading when compared against the PBF plugin
    • colorado-latest.osm.pbf was used for profiling. Note: Takes ~8.8 GB of memory to load the file, no clue how much is needed to render it.
    • PBF plugin (max memory in use: 8779 MB):
      • Time spent loading PBF: 49.5s (3 runs with same time spent)
      • Memory allocations: 30,155,126,741 (average across three runs, ~500 MB spread)
      • CPU cyles: 3061 (average across 3 runs, ~600 CPU cycle spread)
    • Patch (max memory in use: 8783 MB):
      • Time spent loading PBF: 75s (average across 3 runs, min 62s, max 89s)
      • Memory allocations: 30,098,125,773 (average across three runs, ~900 MB spread)
      • CPU cyles: 4650 (average across 3 runs, min 3862, max 5356)

Attachments (1)

22603.patch (95.6 KB ) - added by taylor.smock 16 months ago.

Download all attachments as: .zip

Change History (10)

comment:1 by taylor.smock, 16 months ago

Description: modified (diff)

comment:2 by taylor.smock, 16 months ago

Description: modified (diff)
Milestone: 23.02
Owner: changed from team to taylor.smock
Status: newassigned
Summary: [WIP Patch] Add PBF reading support to JOSM[PATCH] Add PBF reading support to JOSM

by taylor.smock, 16 months ago

Attachment: 22603.patch added

comment:3 by taylor.smock, 14 months ago

Milestone: 23.0223.03

Ticket retargeted after milestone closed

comment:4 by taylor.smock, 13 months ago

Resolution: fixed
Status: assignedclosed

In 18695/josm:

Fix #22603: Add PBF reading support

comment:5 by gaben, 13 months ago

The difference between this and the plugin, that the latter can export as well (plus has dependency on the official protobuf)?

(Noticed a few since xxx)

in reply to:  5 ; comment:6 by taylor.smock, 13 months ago

Replying to gaben:

The difference between this and the plugin, that the latter can export as well (plus has dependency on the official protobuf)?

The primary difference is that no plugin is required for PBF reading. Moving forward, something that we will probably be able to do is allow users to use a local PBF file as a "download" source instead of the OSM server (mostly for people who want to work with largish areas).

Beyond that, JOSM can validate files on the command line, so corporate users will (hopefully) integrate JOSM into their validation pipelines.

(Noticed a few since xxx)

FML. I ran ./scripts/since_xxx.py. Probably the entirely new files, since those get put in the actual default changelist, and then IntelliJ does some stuff to make it appear that they are in a different changeset. And I only ran it against the changeset that I was applying.

in reply to:  6 ; comment:7 by gaben, 13 months ago

Replying to taylor.smock:

The primary difference is that no plugin is required for PBF reading. Moving forward, something that we will probably be able to do is allow users to use a local PBF file as a "download" source instead of the OSM server (mostly for people who want to work with largish areas).

Beyond that, JOSM can validate files on the command line, so corporate users will (hopefully) integrate JOSM into their validation pipelines.

I see, that's a quite useful use case.

Also just noticed, r18695 introduced some ErrorProne [MissingSummary] warnings, so nothing serious.

in reply to:  7 comment:8 by taylor.smock, 13 months ago

Replying to gaben:

Replying to taylor.smock:

Beyond that, JOSM can validate files on the command line, so corporate users will (hopefully) integrate JOSM into their validation pipelines.

I see, that's a quite useful use case.

I really want corporate validation teams to use the same tools as editors for validation purposes, instead of writing their own. It means that (a) any validation check they write will be usable by editor users so that (b) fewer issues are introduced into the OSM database and (c) more feedback can be given on the validations.

As far as the missing summary warnings go, I'll try to fix them this week -- I'm also working on updating the Tasking Manager dependencies, and hopefully I'll finish those in the next day or two.

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The owner will remain taylor.smock.
as The resolution will be set.
The resolution will be deleted. Next status will be 'reopened'.

Add Comment


E-mail address and name can be saved in the Preferences .
 
Note: See TracTickets for help on using tickets.