Modify

Opened 2 years ago

Last modified 12 days ago

#17177 new enhancement

[WIP PATCH] Add support for Mapbox Vector Tile

Reported by: Don-vip Owned by: team
Priority: major Milestone: 21.03
Component: Core imagery Version:
Keywords: mvt mapbox vector tile Cc:

Description

OpenInfraMap dropped its classic tile servers to switch to vector tiles.

If we want to display it in JOSM we must first support MVT format, built on PBF.

Old Maps/Worldwide entries for reference and possible future restoration:

    <entry overlay="true">
        <name>OpenInfraMap Telecoms</name>
        <id>openinframap-telecoms</id>
        <category>osmbasedmap</category>
        <type>tms</type>
        <description lang="en">Overlay imagery for telecoms infrastructure mapping in OSM. Updated daily.</description>
        <url>https://tiles-{switch:a,b,c}.openinframap.org/telecoms/{zoom}/{x}/{y}.png</url>
        <attribution-text>© OpenInfraMap.org</attribution-text>
        <attribution-url>https://www.openinframap.org/</attribution-url>
        <max-zoom>19</max-zoom>
        <valid-georeference>true</valid-georeference>
    </entry>

    <entry overlay="true">
        <name>OpenInfraMap Power</name>
        <id>openinframap-power</id>
        <category>osmbasedmap</category>
        <type>tms</type>
        <description lang="en">Overlay imagery for power infrastructure mapping in OSM. Updated daily.</description>
        <url>https://tiles-{switch:a,b,c}.openinframap.org/power/{zoom}/{x}/{y}.png</url>
        <attribution-text>© OpenInfraMap.org</attribution-text>
        <attribution-url>https://www.openinframap.org/</attribution-url>
        <max-zoom>19</max-zoom>
        <valid-georeference>true</valid-georeference>
    </entry>

    <entry overlay="true">
        <name>OpenInfraMap Petroleum</name>
        <id>openinframap-petroleum</id>
        <category>osmbasedmap</category>
        <type>tms</type>
        <description lang="en">Overlay imagery for petroleum infrastructure mapping in OSM. Updated daily.</description>
        <url>https://tiles-{switch:a,b,c}.openinframap.org/petroleum/{zoom}/{x}/{y}.png</url>
        <attribution-text>© OpenInfraMap.org</attribution-text>
        <attribution-url>https://www.openinframap.org/</attribution-url>
        <max-zoom>19</max-zoom>
        <valid-georeference>true</valid-georeference>
    </entry>

Attachments (4)

17177.patch (37.9 KB) - added by taylor.smock 4 weeks ago.
WIP Patch. DO NOT APPLY. Parsing of Mapbox Vector Tiles partially works. Packed fields do not yet work (specifically, tags and geometry)
17177.1.patch (77.9 KB) - added by taylor.smock 4 weeks ago.
Initial rendering work (currently only works on zoom level 14)
17177.2.patch (82.9 KB) - added by taylor.smock 4 weeks ago.
Rendering works on all zoom levels (quirks are present at low zoom levels)
17177.3.patch (103.8 KB) - added by taylor.smock 2 weeks ago.
Fix rendering issue with regards to a multipolygons being incorrectly rendered. This is probably good enough for initial testing (if this patch gets merged, functionality should be hidden behind expert mode).

Download all attachments as: .zip

Change History (16)

comment:1 Changed 2 years ago by GerdP

I don't see how this is related to the pbf plugin. IIGTR their format just also uses Google protobuf to specify the format.

comment:2 Changed 2 years ago by Don-vip

Component: Plugin pbfCore imagery
Owner: changed from Don-vip to team

comment:3 Changed 5 months ago by pyrog

Will be very useful to display vector tiles in JOSM :)
And to query them too.

iD do this :)
But unfortunately this editor is not powerful as JOSM.

Last edited 5 months ago by pyrog (previous) (diff)

comment:4 Changed 4 weeks ago by taylor.smock

Summary: Add support for Mapbox Vector Tile[RFC] Add support for Mapbox Vector Tile

I'm starting to look at the requirements to implement this.

Questions:

  • Do we want to add com.google.protobuf (~1.6 MB, so a bit more than 10% of the current JOSM size) or attempt to write our own protobuf reader? (See Google encoding docs for encoding documentation, I've started on a basic implementation, but it may be better/easier to use the com.google.protobuf library)
  • If we use com.google.protobuf, do we want to merge the pbf plugin into JOSM core? (I doubt this -- there is another dependency for it)

comment:5 in reply to:  4 ; Changed 4 weeks ago by stoecker

Replying to taylor.smock:

I'm starting to look at the requirements to implement this.

Questions:

  • Do we want to add com.google.protobuf (~1.6 MB, so a bit more than 10% of the current JOSM size) or attempt to write our own protobuf reader? (See Google encoding docs for encoding documentation, I've started on a basic implementation, but it may be better/easier to use the com.google.protobuf library)
  • If we use com.google.protobuf, do we want to merge the pbf plugin into JOSM core? (I doubt this -- there is another dependency for it)

I'd rather say the support should go into a plugin as well when it needs such heavy libraries.

comment:6 in reply to:  5 ; Changed 4 weeks ago by taylor.smock

Summary: [RFC] Add support for Mapbox Vector Tile[WIP PATCH] Add support for Mapbox Vector Tile

Replying to stoecker:

[...]

I'd rather say the support should go into a plugin as well when it needs such heavy libraries.

I've started working on an custom implementation for protobuf. Based off of what I've done so far, I have no idea why the official protobuf library is 1.6 MB. Probably dependencies, TBH. (Maybe Guava?).

Anyway, WIP Patch notes:

Changed 4 weeks ago by taylor.smock

Attachment: 17177.patch added

WIP Patch. DO NOT APPLY. Parsing of Mapbox Vector Tiles partially works. Packed fields do not yet work (specifically, tags and geometry)

comment:7 in reply to:  6 ; Changed 4 weeks ago by stoecker

I've started working on an custom implementation for protobuf. Based off of what I've done so far, I have no idea why the official protobuf library is 1.6 MB. Probably dependencies, TBH. (Maybe Guava?).

Very common effect. Use a library for a 5 line function and another one and another one and ...

If support for protobuf can be that small maybe pbf plugin can go into core as well ;-)

Changed 4 weeks ago by taylor.smock

Attachment: 17177.1.patch added

Initial rendering work (currently only works on zoom level 14)

comment:8 in reply to:  7 Changed 4 weeks ago by taylor.smock

Now I just have to finish (basic) rendering.

I'm having lots of fun trying to figure out why zoom level 14 works, but no other zoom does (I'm figuring I probably have to figure out some AffineTransform for it).

Changed 4 weeks ago by taylor.smock

Attachment: 17177.2.patch added

Rendering works on all zoom levels (quirks are present at low zoom levels)

comment:9 Changed 4 weeks ago by taylor.smock

What still needs to be done:

  • Check rendering on HiDPI monitor (surprisingly, the magic number works on Fedora with Java version: 15.0.2+7, Red Hat, Inc., OpenJDK 64-Bit Server VM and Screen: :0.0 3840×2160 (scaling 1.00×1.00) :0.1 3840×2160 (scaling 1.00×1.00))
  • Extend tests, and check random areas to ensure everything works well
  • Add some kind of method for layers to have arbitrary rendering (this should probably be a new ticket, we should probably support https://docs.mapbox.com/mapbox-gl-js/style-spec )
  • Ability to query data on layer
  • Ability to toggle vector tile layers (not JOSM layers)
  • Cleanup TODOs in patch
  • Add support for MVT layers in imagery index parser (may already be done)
  • Handle overlap of areas better (see OpenInfraMap)

Test layers:

Last edited 2 weeks ago by taylor.smock (previous) (diff)

Changed 2 weeks ago by taylor.smock

Attachment: 17177.3.patch added

Fix rendering issue with regards to a multipolygons being incorrectly rendered. This is probably good enough for initial testing (if this patch gets merged, functionality should be hidden behind expert mode).

comment:10 Changed 2 weeks ago by stoecker

Milestone: 21.02

comment:11 Changed 12 days ago by Don-vip

Milestone: 21.0221.03
Priority: normalmajor

Nice! I'll take a look for next version.

comment:12 Changed 12 days ago by taylor.smock

Thanks. I don't know if you have a HiDPI screen or not, but I've only tested on the following combinations:

  • Fedora 33 with 4K monitors (Java 15)
  • MacOS 10.15 with 1080p and 1800p monitors (Java 11)

I'm worried that the "magic number" I have (32768 or 2^15) only works for specific configurations.

Notes:

  • Quirks are present at low zoom levels (the tiles load and then disappear -- I need to debug this)
  • Only static styling is implemented (so points are green non-filled circles, lines are red, and areas are yellow).
  • Only the base URL is supported -- stylesheets that conform to https://docs.mapbox.com/mapbox-gl-js/style-spec are not yet supported, although they contain both template URLs for the tiles and style information
  • I'm currently debating over whether or not I want to convert tiles to a DataSet and reuse the rendering pipeline for mapcss (this is tempting, I'd probably have to add a few methods for getWays(bbox, zoom) and have some zoom-specific buckets or something, just to reduce cycles)
  • There is no deduplication of features. So if a line or polygon crosses a tile boundary (see https://www.openstreetmap.org/way/666293900 with https://www.openstreetmap.org/way/666293900 for an example).
  • There could probably be some performance enhancements

Modify Ticket

Change Properties
Set your email in Preferences
Action
as new The owner will remain team.
as The resolution will be set.
to The owner will be changed from team to the specified user.
The owner will change to Don-vip
as duplicate The resolution will be set to duplicate.The specified ticket will be cross-referenced with this ticket
The owner will be changed from team to anonymous.

Add Comment


E-mail address and name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.