Modify

Opened 4 months ago

Closed 4 months ago

Last modified 4 months ago

#16973 closed defect (fixed)

MapCSSRendererTest fails on Ubuntu 18.04

Reported by: Don-vip Owned by: team
Priority: major Milestone: 18.12
Component: Unit tests Version:
Keywords: linux ubuntu mapcss Cc: ris, naoliv, michael2402, wiktorn, simon04, floscher

Description (last modified by Don-vip)

As I feared with #16688, org.openstreetmap.josm.gui.mappaint.MapCSSRendererTest does not behave the same on Ubuntu 16.04 and Ubuntu 18.04.

https://josm.openstreetmap.de/jenkins/job/JOSM/lastCompletedBuild/testReport/

I don't know why, but this is problematic as we need it to run on both versions now we have updated to 18.04 but Travis CI is still using 16.04.

Test Reference Result Difference
area-text
eval
node-text
node-text2
way-text

Attachments (15)

area-text-reference.png (5.1 KB) - added by Don-vip 4 months ago.
area-text-test-differences.png (657 bytes) - added by Don-vip 4 months ago.
area-text-test-output.png (5.4 KB) - added by Don-vip 4 months ago.
eval-reference.png (5.3 KB) - added by Don-vip 4 months ago.
eval-test-differences.png (2.8 KB) - added by Don-vip 4 months ago.
eval-test-output.png (5.7 KB) - added by Don-vip 4 months ago.
node-text-reference.png (6.5 KB) - added by Don-vip 4 months ago.
node-text-test-differences.png (1.1 KB) - added by Don-vip 4 months ago.
node-text-test-output.png (7.0 KB) - added by Don-vip 4 months ago.
node-text2-reference.png (2.3 KB) - added by Don-vip 4 months ago.
node-text2-test-differences.png (610 bytes) - added by Don-vip 4 months ago.
node-text2-test-output.png (2.8 KB) - added by Don-vip 4 months ago.
way-text-reference.png (20.7 KB) - added by Don-vip 4 months ago.
way-text-test-differences.png (672 bytes) - added by Don-vip 4 months ago.
way-text-test-output.png (20.7 KB) - added by Don-vip 4 months ago.

Download all attachments as: .zip

Change History (30)

comment:1 Changed 4 months ago by michael2402

Do we know which JVM versions were used in both cases? I think I noticed differences between SUN and OpenJDK some time in the past.

comment:2 Changed 4 months ago by ris

I've always had problems with these tests locally FWIW.

comment:3 Changed 4 months ago by michael2402

@ris Wich JVM are you using and which OS?

The problem seems to be that they differ in rendering and the tests are doing a hard comparison of the images. We could add a threshold but judging visual similarity using a simple algorithm is difficult.

comment:4 Changed 4 months ago by Don-vip

Last edited 4 months ago by Don-vip (previous) (diff)

comment:5 Changed 4 months ago by michael2402

I will test it

Does anyone know the ant equivalent to the gradle --tests parameter (that only executes specific tests)?

comment:6 Changed 4 months ago by ris

OpenJDK 8, linux.

@michael2402 incidentally I added a tool to the testutils recently for making lenient checks on simple image content - ImagePatternMatching: https://github.com/openstreetmap/josm/blob/46e9e88b6de60be69bba0e8615d93c85222fef89/test/unit/org/openstreetmap/josm/testutils/ImagePatternMatching.java

Essentially allows you to make assertions about strips of the image using regular expressions.

comment:7 in reply to:  5 Changed 4 months ago by Don-vip

Replying to michael2402:

Does anyone know the ant equivalent to the gradle --tests parameter (that only executes specific tests)?

It's a bit complicated right now, see wiki/DevelopersGuide/Compiling

It used to be easy from Eclipse (right click on test, "run as Junit") but currently more difficult, see #17012

comment:8 Changed 4 months ago by Don-vip

Milestone: 18.1118.12

comment:9 Changed 4 months ago by Don-vip

Cc: wiktorn simon04 floscher added

@team does anyone have a lead to fix the issue? The build is failing since we updated the server to latest version of Ubuntu.

comment:10 Changed 4 months ago by Don-vip

Priority: normalmajor

comment:11 Changed 4 months ago by stoecker

When the test differs for Java version why not add different reference test data after verifying that the new data is only different, but not wrong?

comment:12 Changed 4 months ago by michael2402

We can just re-generate all data using the new configuration. Then visually check if it looks OK. Those are just problems with anti-aliasing and some rounding issues that are one pixel of.

The test is mostly there to be able to see if some map features just got "lost" during a refactoring (e.g. labels not being rendered, ...). For that, we don't need to be able to run it in every environment, although it would be nice.

Changed 4 months ago by Don-vip

Attachment: area-text-reference.png added

Changed 4 months ago by Don-vip

Changed 4 months ago by Don-vip

Attachment: area-text-test-output.png added

Changed 4 months ago by Don-vip

Attachment: eval-reference.png added

Changed 4 months ago by Don-vip

Attachment: eval-test-differences.png added

Changed 4 months ago by Don-vip

Attachment: eval-test-output.png added

Changed 4 months ago by Don-vip

Attachment: node-text-reference.png added

Changed 4 months ago by Don-vip

Changed 4 months ago by Don-vip

Attachment: node-text-test-output.png added

Changed 4 months ago by Don-vip

Attachment: node-text2-reference.png added

Changed 4 months ago by Don-vip

Changed 4 months ago by Don-vip

Attachment: node-text2-test-output.png added

Changed 4 months ago by Don-vip

Attachment: way-text-reference.png added

Changed 4 months ago by Don-vip

Changed 4 months ago by Don-vip

Attachment: way-text-test-output.png added

comment:13 Changed 4 months ago by Don-vip

Description: modified (diff)

comment:14 Changed 4 months ago by Don-vip

Resolution: fixed
Status: newclosed

In 14491/josm:

fix #16973 - update mapcss reference images for text to match output we get on Ubuntu 18.04

comment:15 Changed 4 months ago by ris

I don't think it's ever a good idea to expect pixel-perfect results from a test unless you're in control of the whole rendering process. In this case we defer the font rendering to swing, which in turn defers it I imagine to a platform-dependent font renderer, in this case freetype.

Tying the test to bit-perfect results will also make a test platform-dependent and probably making local results or travis results rather valueless. I've done this using my ImagePatternMatching tool, but in this case it might be more useful to use a "perceptual" diff or hash like e.g. http://pdiff.sourceforge.net/ or https://www.phash.org/

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The owner will remain team.
as The resolution will be set.
The resolution will be deleted.

Add Comment


E-mail address and name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.