Opened 12 years ago

Closed 12 years ago

#7740 closed enhancement (fixed)

[PATCH] Fix roundToOsmPrecision() accuracy

Reported by: mrwojo Owned by: team
Priority: normal Milestone:
Component: Core Version:
Keywords: precision rounding Cc:


Currently, LatLon.roundToOsmPrecision(double) fails to pass checks in LatLonTest.roundingTest() due to rounding issues noted in roundToOsmPrecision(). This patch should address the accuracy issues by simply inverting the operations (multiply by 1e7 instead of divide by 1e-7 etc.). Errors creep in when dividing and multiplying by 1e-7 because that value cannot be represented accurately in standard floating point.

This should make roundToOsmPrecision(double) pass all the tests in LatLonTest.roundingTest() without affecting performance. It even passes tests with small input values, like:

        assertEquals(LatLon.roundToOsmPrecision(0.02345678),  0.0234568, 0);

(roundToOsmPrecisionStrict(double) leaves more than 7 digits of precision in near-zero values.)

Since it passes the rounding tests (and more), it could make roundToOsmPrecisionStrict(double) redundant.

Attachments (1)

roundToOsmPrecision.patch (2.1 KB ) - added by mrwojo 12 years ago.
also made roundToOsmPrecisionStrict() call roundToOsmPrecision()

Download all attachments as: .zip

Change History (3)

comment:1 by simon04, 12 years ago

Nice. Will be considered after end of stabilization phase (DevelopersGuide/Schedule).

What about implementing roundToOsmPrecisionStrict in terms of roundToOsmPrecision and changing the documentation accordingly?

by mrwojo, 12 years ago

Attachment: roundToOsmPrecision.patch added

also made roundToOsmPrecisionStrict() call roundToOsmPrecision()

comment:2 by simon04, 12 years ago

Resolution: fixed
Status: newclosed

In 5268/josm:

fix #7740 - Fix roundToOsmPrecision() accuracy (patch by mrwojo)

Modify Ticket

Change Properties
Set your email in Preferences
as closed The owner will remain team.
as The resolution will be set.
The resolution will be deleted. Next status will be 'reopened'.

Add Comment

E-mail address and name can be saved in the Preferences .
Note: See TracTickets for help on using tickets.