JCSCachedTileLoaderJobTest is failing

[bastiK]

The unit test JCSCachedTileLoaderJobTest is failing in Jenkins. (Does work on my local machine.)

[anonymous]

Yes, I'm investigating this from some time. I can only reproduce this when running tests on full blown Ubuntu via ant. Running it on debian/jessie via Docker or Alpine Linux via Docker, directly on Windows - it all passes.

No chances though for attaching debugger.

[bastiK]

You can restrict the tests that run via ant like so:

build.xml

@@ -439 +439 @@
                     <formatter type="plain"/>
                     <formatter type="xml"/>
                     <batchtest fork="yes" todir="${test.dir}/report">
-                        <fileset dir="${test.dir}/build/@{testfamily}" includes="**/*Test@{testITsuffix}.class"/>
+                        <fileset dir="${test.dir}/build/@{testfamily}" includes="**/*JCSCachedTileLoaderJobTest@{testITsuffix}.class"/>
                     </batchtest>
                 </junit>
             </jacoco:coverage>

If it passes like this, you can figure out which preceding test causes it to fail.

[wiktorn]

Been there. This change to ant file prevents triggering the bug :-o

Looks a bit like a ​Heisenbug

[bastiK]

The idea is to make a long list of all available tests and copy & paste them into the ant file like so:

build.xml

@@ -439 +439 @@
                     <formatter type="plain"/>
                     <formatter type="xml"/>
                     <batchtest fork="yes" todir="${test.dir}/report">
-                        <fileset dir="${test.dir}/build/@{testfamily}" includes="**/*Test@{testITsuffix}.class"/>
+                        <fileset dir="${test.dir}/build/@{testfamily}">
+                            <include name="**/org/openstreetmap/josm/data/BoundariesTestIT.class"/>
+                            <include name="**/org/openstreetmap/josm/data/imagery/ImageryCompareTestIT.class"/>
+                            ...
+                        </fileset>
                     </batchtest>
                 </junit>
             </jacoco:coverage>

If the order is the same as in the wildcard run, the bug will trigger. Then you can start removing other tests until you find the one that is the trigger.

I did this once, it's a lot of fun ...

[wiktorn]

In 11473/josm:

Always clear cache before tests.

Closes: #14255

[wiktorn]

Thanks for help bastiK.

I used this part to extract the exact order, in which tests are run

<path id="debug">
        <fileset dir="${test.dir}/build/@{testfamily}" includes="**/*Test@{testITsuffix}.class"/>
</path>
<pathconvert property="echo.path.compile" refid="debug" pathsep="${line.separator}"></pathconvert>
<echo>${echo.path.compile}</echo>

And then set my junit fileset to (after identifying interlocked tests):

<batchtest fork="yes" todir="${test.dir}/report">
                        <fileset dir="${test.dir}/build/@{testfamily}/">
                                <include name="**/org/openstreetmap/josm/data/cache/HostLimitQueueTest@{testITsuffix}.class" /> 
                                <include name="**/org/openstreetmap/josm/data/cache/JCSCachedTileLoaderJobTest@{testITsuffix}.class" />
</fileset>