Opened 10 years ago

Closed 10 years ago

#7740 closed enhancement (fixed)

[PATCH] Fix roundToOsmPrecision() accuracy

Reported by: mrwojo Owned by: team
Priority: normal Milestone:
Component: Core Version:
Keywords: precision rounding Cc:


Currently, LatLon.roundToOsmPrecision(double) fails to pass checks in LatLonTest.roundingTest() due to rounding issues noted in roundToOsmPrecision(). This patch should address the accuracy issues by simply inverting the operations (multiply by 1e7 instead of divide by 1e-7 etc.). Errors creep in when dividing and multiplying by 1e-7 because that value cannot be represented accurately in standard floating point.

This should make roundToOsmPrecision(double) pass all the tests in LatLonTest.roundingTest() without affecting performance. It even passes tests with small input values, like:

        assertEquals(LatLon.roundToOsmPrecision(0.02345678),  0.0234568, 0);

(roundToOsmPrecisionStrict(double) leaves more than 7 digits of precision in near-zero values.)

Since it passes the rounding tests (and more), it could make roundToOsmPrecisionStrict(double) redundant.

Attachments (1)

roundToOsmPrecision.patch (2.1 KB) - added by mrwojo 10 years ago.
also made roundToOsmPrecisionStrict() call roundToOsmPrecision()

Download all attachments as: .zip

Change History (3)

comment:1 Changed 10 years ago by simon04

Nice. Will be considered after end of stabilization phase (DevelopersGuide/Schedule).

What about implementing roundToOsmPrecisionStrict in terms of roundToOsmPrecision and changing the documentation accordingly?

Changed 10 years ago by mrwojo

Attachment: roundToOsmPrecision.patch added

also made roundToOsmPrecisionStrict() call roundToOsmPrecision()

comment:2 Changed 10 years ago by simon04

Resolution: fixed
Status: newclosed

In 5268/josm:

fix #7740 - Fix roundToOsmPrecision() accuracy (patch by mrwojo)

Modify Ticket

Change Properties
Set your email in Preferences
as closed The owner will remain team.
as The resolution will be set.
The resolution will be deleted.

Add Comment

E-mail address and name can be saved in the Preferences.

Note: See TracTickets for help on using tickets.