Modify

Opened 6 years ago

Last modified 3 years ago

#16860 reopened enhancement

Resolve all dependencies and tools using Apache Ivy

Reported by: wiktorn Owned by: team
Priority: major Milestone: Longterm
Component: Core Version:
Keywords: hack-weekend-2018-10 Cc: stoecker, sebastic

Description (last modified by wiktorn)

Follow up to: #8269 and #16420. Needed for: #16858

Use JMapViewer from our Nexus instead of use of svn:externals. This allows us to:

  • test in the field, how well does it's job in controlling our dependencies
  • move JMapViewer outside OSM SVN server

The goal is to do incremental changes and provide better dependency management than following HEAD on svn:externals.

Sponsored by: OpenStreetMap Hack Weekend, Karlsruhe Oct'18

Attachments (2)

ivy-jmapviewer.patch (13.8 KB ) - added by wiktorn 6 years ago.
ivy-jmapviewer-v2.patch (12.3 KB ) - added by simon04 6 years ago.

Download all attachments as: .zip

Change History (170)

by wiktorn, 6 years ago

Attachment: ivy-jmapviewer.patch added

comment:1 by wiktorn, 6 years ago

Description: modified (diff)
Keywords: hack-weekend-2018-10 added

comment:2 by Don-vip, 6 years ago

Milestone: 18.1018.11

comment:3 by Don-vip, 6 years ago

Milestone: 18.1118.12

comment:4 by Don-vip, 6 years ago

Milestone: 18.1219.01

by simon04, 6 years ago

Attachment: ivy-jmapviewer-v2.patch added

comment:5 by simon04, 6 years ago

I merged the patch with the recent changes and uploaded attachment:ivy-jmapviewer-v2.patch

Is there anything holding us back?

in reply to:  5 comment:6 by stoecker, 6 years ago

Is there anything holding us back?

Yes. The new server. We don't want to install and do anything new on the old one.

comment:7 by Don-vip, 6 years ago

Milestone: 19.0119.02

comment:8 by Don-vip, 6 years ago

Milestone: 19.0219.03

comment:9 by Don-vip, 6 years ago

Milestone: 19.0319.04

comment:10 by Don-vip, 6 years ago

Milestone: 19.0419.05

comment:11 by Don-vip, 6 years ago

Milestone: 19.05

comment:12 by simon04, 5 years ago

Milestone: 20.02

Since we got a new server in the meanwhile, is anything (else) holding us back?

in reply to:  12 comment:13 by Don-vip, 5 years ago

Replying to simon04:

is anything (else) holding us back?

I don't know yet how to complete the GitLab setup. I'll comment why in #16857 comments.

comment:14 by simon04, 5 years ago

Milestone: 20.0220.03

comment:15 by simon04, 5 years ago

In 15977/josm:

see #16860, see #18140 - Setup Apache Ivy (patch by wiktorn, modified)

comment:16 by simon04, 5 years ago

In 16011/josm:

see #16860, see #18140 - Restrict SpotBugs to org/openstreetmap/josm

comment:17 by Don-vip, 5 years ago

In 16020/josm:

see #16860, see #18140 - better way to restrict SpotBugs analysis scope (without error about missing classes)

See https://github.com/spotbugs/spotbugs/blob/4.0.0/spotbugs-ant/src/main/java/edu/umd/cs/findbugs/anttask/FindBugsTask.java

comment:18 by simon04, 5 years ago

In 16021/josm:

see #16860 - Resolve org.glassfish:javax.json via Apache Ivy

comment:19 by simon04, 5 years ago

In 16022/josm:

see #16860 - Resolve org.tukaani:xz via Apache Ivy

comment:20 by simon04, 5 years ago

In 16023/josm:

see #16860 - Resolve org.apache.commons:commons-compress via Apache Ivy

comment:21 by simon04, 5 years ago

In 16024/josm:

see #16860 - Resolve commons-logging:commons-logging via Apache Ivy

comment:22 by simon04, 5 years ago

In 16025/josm:

see #16860 - Resolve com.drewnoakes:metadata-extractor via Apache Ivy

in reply to:  22 ; comment:23 by Don-vip, 5 years ago

Replying to simon04:

In 16025/josm:

see #16860 - Resolve com.drewnoakes:metadata-extractor via Apache Ivy

Are you sure it works for this one? I remember I must patch it at each update.

comment:24 by simon04, 5 years ago

Here's how the dist sizes evolved:

ls -l --time-style full-iso 
total 84764
-rw-r--r-- 1 simon users 14443614 2020-03-03 22:21:13 +0100 josm-custom-16016.jar
-rw-r--r-- 1 simon users 14444368 2020-03-03 22:23:10 +0100 josm-custom-16016+json.jar
-rw-r--r-- 1 simon users 14429230 2020-03-03 22:24:45 +0100 josm-custom-16016+json+xz.jar
-rw-r--r-- 1 simon users 14432267 2020-03-03 22:37:35 +0100 josm-custom-16016+json+xz+compress.jar
-rw-r--r-- 1 simon users 14430688 2020-03-03 22:40:41 +0100 josm-custom-16016+json+xz+compress+logging.jar
-rw-r--r-- 1 simon users 14604031 2020-03-03 23:28:36 +0100 josm-custom-16016+json+xz+compress+logging+metadata-extractor.jar

The increase for metadata-extractor is due to our disabled imports in JpegMetadataReader.java which cannot be ported to Apache Ivy.


How to proceed?

in reply to:  23 ; comment:25 by simon04, 5 years ago

Replying to Don-vip:

Are you sure it works for this one? I remember I must patch it at each update.

Haven't found any problem yet. I successfully opened a bunch of JPG images; ExifReaderTest passes.

in reply to:  24 comment:26 by Don-vip, 5 years ago

Replying to simon04:

The increase for metadata-extractor is due to our disabled imports in JpegMetadataReader.java which cannot be ported to Apache Ivy.

We could ask the author to provide smaller artifacts.

in reply to:  24 ; comment:27 by Don-vip, 5 years ago

Replying to simon04:

  • signpost: we did some minor cleanups and kicked out commons-codecs in r8149, r10746

Signpost is dead, no commit in 5 years, 15 open pull requests...

Did we consider https://github.com/scribejava/scribejava at some point? It looks way more active, but probably heavier.

in reply to:  25 ; comment:28 by Don-vip, 5 years ago

Replying to simon04:

Replying to Don-vip:

Are you sure it works for this one? I remember I must patch it at each update.

Haven't found any problem yet. I successfully opened a bunch of JPG images; ExifReaderTest passes.

I'm almost sure it doesn't work. Look at proguard output:

 [proguard] Warning: com.drew.imaging.FileTypeDetector: can't find referenced class com.drew.imaging.quicktime.QuickTimeTypeChecker
 [proguard] Warning: com.drew.imaging.FileTypeDetector: can't find referenced class com.drew.imaging.quicktime.QuickTimeTypeChecker
 [proguard] Warning: com.drew.imaging.FileTypeDetector: can't find referenced class com.drew.imaging.riff.RiffTypeChecker
 [proguard] Warning: com.drew.imaging.FileTypeDetector: can't find referenced class com.drew.imaging.riff.RiffTypeChecker
 [proguard] Warning: com.drew.imaging.FileTypeDetector: can't find referenced class com.drew.imaging.mp3.MpegAudioTypeChecker
 [proguard] Warning: com.drew.imaging.FileTypeDetector: can't find referenced class com.drew.imaging.mp3.MpegAudioTypeChecker
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.psd.PsdMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.png.PngMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.bmp.BmpMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.gif.GifMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.ico.IcoMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.pcx.PcxMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.webp.WebpMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.raf.RafMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.avi.AviMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.wav.WavMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.quicktime.QuickTimeMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.mp4.Mp4MetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.mp3.Mp3MetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.eps.EpsMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.heif.HeifMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.psd.PsdMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.png.PngMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.bmp.BmpMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.gif.GifMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.ico.IcoMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.pcx.PcxMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.webp.WebpMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.raf.RafMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.avi.AviMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.wav.WavMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.quicktime.QuickTimeMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.mp4.Mp4MetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.mp3.Mp3MetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.eps.EpsMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.heif.HeifMetadataReader
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$1: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamFactory
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$1: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamFactory
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$1: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamFactory
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$1: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamFactory
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$1: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamProvider
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$1: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamProvider
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$1: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamFactory
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$1: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamProvider
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$1: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamProvider
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$1: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamProvider
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$1: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamProvider
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$1: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamFactory
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$2: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamFactory
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$2: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamFactory
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$2: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamFactory
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$2: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamFactory
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$2: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamProvider
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$2: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamProvider
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$2: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamFactory
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$2: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamProvider
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$2: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamProvider
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$2: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamProvider
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$2: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamProvider
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$2: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamFactory
 [proguard] Warning: org.apache.commons.compress.compressors.xz.XZUtils$CachedAvailability: can't find referenced class org.apache.commons.compress.compressors.xz.XZUtils
 [proguard] Warning: there were 61 unresolved references to classes or interfaces.

in reply to:  27 comment:29 by simon04, 5 years ago

Replying to Don-vip:

Did we consider https://github.com/scribejava/scribejava at some point? It looks way more active, but probably heavier.

Half an hour ago, I've also found this library. Looks interesting. Will investigate in the next days. The first thing I noticed is that com.github.scribejava.core.extractors.AbstractOAuth1JSONTokenExtractor makes use of Jackson for JSON decoding, but we could probably provide our own implementation.

in reply to:  28 comment:30 by simon04, 5 years ago

Replying to Don-vip:

I'm almost sure it doesn't work. Look at proguard output:

This is the list of classes which are now included in josm.jar, but haven't been before:

$ comm -3 all-classes-in-josm.jar-before all-classes-in-josm.jar-after
        com/adobe/
        com/adobe/internal/
        com/adobe/internal/xmp/
        com/adobe/internal/xmp/impl/
        com/adobe/internal/xmp/impl/Base64.class
        com/adobe/internal/xmp/impl/ByteBuffer.class
        com/adobe/internal/xmp/impl/CountOutputStream.class
        com/adobe/internal/xmp/impl/FixASCIIControlsReader.class
        com/adobe/internal/xmp/impl/ISO8601Converter.class
        com/adobe/internal/xmp/impl/Latin1Converter.class
        com/adobe/internal/xmp/impl/ParameterAsserts.class
        com/adobe/internal/xmp/impl/ParseRDF.class
        com/adobe/internal/xmp/impl/ParseState.class
        com/adobe/internal/xmp/impl/QName.class
        com/adobe/internal/xmp/impl/Utils.class
        com/adobe/internal/xmp/impl/XMPDateTimeImpl.class
        com/adobe/internal/xmp/impl/XMPIteratorImpl$NodeIterator$1.class
        com/adobe/internal/xmp/impl/XMPIteratorImpl$NodeIteratorChildren.class
        com/adobe/internal/xmp/impl/XMPIteratorImpl$NodeIterator.class
        com/adobe/internal/xmp/impl/XMPIteratorImpl.class
        com/adobe/internal/xmp/impl/XMPMetaImpl$1.class
        com/adobe/internal/xmp/impl/XMPMetaImpl$2.class
        com/adobe/internal/xmp/impl/XMPMetaImpl.class
        com/adobe/internal/xmp/impl/XMPMetaParser.class
        com/adobe/internal/xmp/impl/XMPNode$1.class
        com/adobe/internal/xmp/impl/XMPNode.class
        com/adobe/internal/xmp/impl/XMPNodeUtils.class
        com/adobe/internal/xmp/impl/XMPNormalizer.class
        com/adobe/internal/xmp/impl/XMPSchemaRegistryImpl$1.class
        com/adobe/internal/xmp/impl/XMPSchemaRegistryImpl.class
        com/adobe/internal/xmp/impl/XMPSerializerHelper.class
        com/adobe/internal/xmp/impl/XMPSerializerRDF.class
        com/adobe/internal/xmp/impl/XMPUtilsImpl.class
        com/adobe/internal/xmp/impl/xpath/
        com/adobe/internal/xmp/impl/xpath/PathPosition.class
        com/adobe/internal/xmp/impl/xpath/XMPPath.class
        com/adobe/internal/xmp/impl/xpath/XMPPathParser.class
        com/adobe/internal/xmp/impl/xpath/XMPPathSegment.class
        com/adobe/internal/xmp/options/
        com/adobe/internal/xmp/options/AliasOptions.class
        com/adobe/internal/xmp/options/IteratorOptions.class
        com/adobe/internal/xmp/options/Options.class
        com/adobe/internal/xmp/options/ParseOptions.class
        com/adobe/internal/xmp/options/PropertyOptions.class
        com/adobe/internal/xmp/options/SerializeOptions.class
        com/adobe/internal/xmp/properties/
        com/adobe/internal/xmp/properties/XMPAliasInfo.class
        com/adobe/internal/xmp/properties/XMPProperty.class
        com/adobe/internal/xmp/properties/XMPPropertyInfo.class
        com/adobe/internal/xmp/utils/
        com/adobe/internal/xmp/utils/XMLStreamWriterFactory.class
        com/adobe/internal/xmp/utils/XMLStreamWriterImpl.class
        com/adobe/internal/xmp/XMPConst.class
        com/adobe/internal/xmp/XMPDateTime.class
        com/adobe/internal/xmp/XMPDateTimeFactory.class
        com/adobe/internal/xmp/XMPError.class
        com/adobe/internal/xmp/XMPException.class
        com/adobe/internal/xmp/XMPIterator.class
        com/adobe/internal/xmp/XMPMeta.class
        com/adobe/internal/xmp/XMPMetaFactory$1.class
        com/adobe/internal/xmp/XMPMetaFactory.class
        com/adobe/internal/xmp/XMPPathFactory.class
        com/adobe/internal/xmp/XMPSchemaRegistry.class
        com/adobe/internal/xmp/XMPUtils.class
        com/adobe/internal/xmp/XMPVersionInfo.class
        com/drew/imaging/FileType.class
        com/drew/imaging/FileTypeDetector.class
        com/drew/imaging/ImageMetadataReader$1.class
        com/drew/imaging/ImageMetadataReader.class
        com/drew/imaging/tiff/TiffMetadataReader.class
        com/drew/imaging/TypeChecker.class
        com/drew/lang/ByteConvert.class
        com/drew/lang/ByteTrie$ByteTrieNode.class
        com/drew/lang/ByteTrie.class
        com/drew/lang/ByteUtil.class
        com/drew/lang/Iterables.class
        com/drew/lang/KeyValuePair.class
        com/drew/lang/NullOutputStream.class
        com/drew/lang/RandomAccessFileReader.class
        com/drew/lang/RandomAccessStreamReader.class
        com/drew/lang/StreamUtil.class
        com/drew/metadata/adobe/
        com/drew/metadata/adobe/AdobeJpegDescriptor.class
        com/drew/metadata/adobe/AdobeJpegDirectory.class
        com/drew/metadata/adobe/AdobeJpegReader.class
        com/drew/metadata/file/FileTypeDescriptor.class
        com/drew/metadata/file/FileTypeDirectory.class
        com/drew/metadata/jfif/
        com/drew/metadata/jfif/JfifDescriptor.class
        com/drew/metadata/jfif/JfifDirectory.class
        com/drew/metadata/jfif/JfifReader.class
        com/drew/metadata/jfxx/
        com/drew/metadata/jfxx/JfxxDescriptor.class
        com/drew/metadata/jfxx/JfxxDirectory.class
        com/drew/metadata/jfxx/JfxxReader.class
        com/drew/metadata/Schema.class
        com/drew/metadata/xmp/
        com/drew/metadata/xmp/XmpDescriptor.class
        com/drew/metadata/xmp/XmpDirectory.class
        com/drew/metadata/xmp/XmpReader.class
        com/drew/metadata/xmp/XmpWriter.class

In particular, FileTypeDetector and ImageMetadataReader are present in this list. We could exclude all those …

Last edited 5 years ago by simon04 (previous) (diff)

comment:31 by simon04, 5 years ago

In 16026/josm:

see #16860 - Apache Ivy: exclude various subclasses from inclusion

comment:32 by simon04, 5 years ago

In 16027/josm:

see #16860 - Apache Ivy: exclude com/drew/imaging/{FileTypeDetector,ImageMetadataReader} from inclusion

comment:33 by Don-vip, 5 years ago

In 16050/josm:

see #18845 - see #16860 - directory cleanup

in reply to:  27 comment:34 by simon04, 5 years ago

Replying to Don-vip:

Replying to simon04:

  • signpost: we did some minor cleanups and kicked out commons-codecs in r8149, r10746

Signpost is dead, no commit in 5 years, 15 open pull requests...

I posted to the signpost mailing list: https://groups.google.com/forum/#!msg/signpost-users/QDccoNNTvpc/5pRpdLZyAAAJ

comment:35 by simon04, 5 years ago

Cc: sebastic added

Hi sebastic, this might affect the packaging for Debian. A few libraries are obtained as JAR via Apache Ivy, see source:trunk/ivy.xml

comment:36 by sebastic, 5 years ago

move JMapViewer outside OSM SVN server

Where is it going? There is no JMapViewer repo on GitHub yet.

The Debian package uses the ZIP releases from https://svn.openstreetmap.org/applications/viewer/jmapviewer/releases/.

In the ivy config I see metadata-extractor, we used to build the josm package with the separately packaged metadata-extractor but because its API is highly unstable updating always broke either josm or one of the other dependent packages.

If the sources for unstable dependencies like this won't be bundled with josm any more, we won't be able to keep to the josm package in Debian.

comment:37 by simon04, 5 years ago

The goal is to use the official JAR distributions for external dependencie, and not (partially) compile them within the JOSM build system.

The current status:

  • Apache Commons Compress ✔
  • Apache Commons JCS: TODO, needs new release, see comment:24
  • Apache Commons Logging ✔
  • SvgSalamander: UNLIKELY
  • Metadata Extractor ✔
  • Signpost: TODO, WIP, see comment:34
  • Jsonp ✔
  • OpeningHoursValidator ✔
  • JMapViewer: TODO (needs separate repository first)

Running ant dist depends on ant resolve (which runs Apache Ivy to fetch the dependencies) and bundles the dependencies in a fatjar. So, by default all dependencies are bundled into dist/josm-custom.jar.

comment:38 by sebastic, 5 years ago

ant resolve doesn't work in a Debian build environment as it doesn't have network, the dependencies need to have their source included in the source tree or an appropriate version needs to be packaged and installed in the build environment.

Experience has shown that using separately packaged metadata-extractor and svgsalamander made backporting JOSM tested for Debian stable a PITA. Once the embedded copies were used this was much less of a problem. The separately packaged JMapViewer has also broken freeplane on more than one occasion, forcing stable users to choose whether to have the josm backport installed (which requires a newer jmapviewer) or a working freeplane.

Because these recent changes make building JOSM from source in a Debian build environment unfeasible when backports are required as well, we'll have to remove the josm package from Debian and replace it with an installer that downloads the pre-built JAR. This is not very good either (because the JARs don't have hashes nor cryptographic signatures to verify the download with), but at least it will keep JOSM tested easily available to users of Debian stable (assuming the josm-installer package passes FTP master review).

comment:39 by simon04, 5 years ago

Is there a Debian compatible way not involving svn:externals? By no means, the goal was to make Debian packging harder/impossible. However, slowly getting rid of svn:externals allows us to move forward in the long-term goal #16871.

comment:40 by sebastic, 5 years ago

Publish a source jar that bundles the required versions of the dependencies?

comment:41 by simon04, 5 years ago

The josm-sources.jar would contain all JOSM's own sources (.java files + resources) and all of its dependencies (plenty of additional .java files + resources)? This seems doable, but requires some investigations into Ant/Ivy (at least, I don't know how to accomplish this by heart).

in reply to:  38 ; comment:42 by wiktorn, 5 years ago

Replying to sebastic:

Because these recent changes make building JOSM from source in a Debian build environment unfeasible when backports are required as well, we'll have to remove the josm package from Debian and replace it with an installer that downloads the pre-built JAR. This is not very good either (because the JARs don't have hashes nor cryptographic signatures to verify the download with), but at least it will keep JOSM tested easily available to users of Debian stable (assuming the josm-installer package passes FTP master review).

Jars published on JOSM page are signed and it is possibile to verify the signature. See: https://docs.oracle.com/en/java/javase/11/tools/jarsigner.html

Replying to simon04:

The josm-sources.jar would contain all JOSM's own sources (.java files + resources) and all of its dependencies (plenty of additional .java files + resources)? This seems doable, but requires some investigations into Ant/Ivy (at least, I don't know how to accomplish this by heart).

If I remember correctly, it was possibile to download sources in Ivy

comment:43 by Don-vip, 5 years ago

Another Ivy issue we need to fix: proxy support, see https://lists.openstreetmap.org/pipermail/josm-dev/2020-March/008288.html

in reply to:  42 comment:44 by sebastic, 5 years ago

Replying to wiktorn:

Jars published on JOSM page are signed and it is possibile to verify the signature. See: https://docs.oracle.com/en/java/javase/11/tools/jarsigner.html

That's not a equivalent to detached GPG signatures.

jarsigner is part of the JDK, installing the entire JDK to run JOSM is highly undesirable.

comment:45 by Don-vip, 5 years ago

In 16118/josm:

see #16860 - instruct Eclipse IvyDE plugin to use ivysettings.xml

comment:46 by Don-vip, 5 years ago

In 16125/josm:

see #16860 - see #18880 - explicit use of Ivy configuration files for Netbeans compatibility

comment:47 by Don-vip, 5 years ago

In 16139/josm:

see #16860 - see #18880 - setup Ivy in Netbeans project

comment:48 by simon04, 5 years ago

In 16141/josm:

see #16860 - build.xml: add ant sources

Generates jar file of JOSM source files and its dependencies.

comment:49 by simon04, 5 years ago

I've asked the apache-jcs maintainers to craft a new release which we could use via Apache Ivy: [JCS-204] Craft a new 2.3 release - https://issues.apache.org/jira/browse/JCS-204

in reply to:  48 ; comment:50 by Don-vip, 5 years ago

Replying to simon04:

In 16141/josm:

see #16860 - build.xml: add ant sources

Generates jar file of JOSM source files and its dependencies.

@sebastic: I guess you don't need the REVISION file?

@simon: Dropping the "create-revision" dependency would make it easier for me to include this new task in the server Makefile

in reply to:  50 comment:51 by sebastic, 5 years ago

Replying to Don-vip:

@sebastic: I guess you don't need the REVISION file?

The package already creates it itself:

https://salsa.debian.org/debian-gis-team/josm/-/blob/debian/0.0.svn15937+dfsg-1/debian/rules#L80

And it's patched to include Build-Name & Debian-Release:

https://salsa.debian.org/debian-gis-team/josm/-/blob/debian/0.0.svn15937+dfsg-1/debian/patches/00-build.patch#L39

comment:52 by simon04, 5 years ago

In 16150/josm:

see #16860 - build.xml: ant sources: remove REVISION

comment:53 by Don-vip, 5 years ago

@sebastic can you please check if https://josm.openstreetmap.de/download/josm-latest-sources.jar is enough for you to build a JOSM Debian package.

comment:54 by sebastic, 5 years ago

JOSM tested 15937 contained these subdirectories under src/ which are not included in the source JAR:

  • javax/json
  • org/jdesktop

comment:55 by simon04, 5 years ago

In 16157/josm:

see #16860 - build.xml: ant sources: use javax.json:javax.json-api

comment:56 by simon04, 5 years ago

  • org/jdesktop is obsolete due to r16050

comment:57 by simon04, 5 years ago

In 16158/josm:

see #16860 - Apache Ivy: re-add org.glassfish:javax.json

comment:58 by Don-vip, 5 years ago

Summary: [PATCH] Remove svn:external for jmapviewer and use Apache Ivy to download jarRemove svn:external for jmapviewer and use Apache Ivy to download jar

comment:59 by simon04, 5 years ago

In 16165/josm:

see #16860 - Resolve PMD using Apache Ivy

comment:60 by Don-vip, 5 years ago

@Simon: if you begin to resolve tools via Ivy, please keep error_prone, no official version actually works, a lot of patches are needed. I'm also using a Jacoco snapshot (unpatched) for Java 15 compatibility, so please wait the next release for this one. All others can be retrieved from Ivy I think.

comment:61 by Don-vip, 5 years ago

Also: don't forget plugins. build-common.xml has references to these jar files.

comment:62 by Don-vip, 5 years ago

@sebastic: I guess you don't need any of these test/static analysis tools for Debian packaging?

comment:63 by simon04, 5 years ago

Summary: Remove svn:external for jmapviewer and use Apache Ivy to download jarResolve all dependencies and tools using Apache Ivy

in reply to:  60 comment:64 by simon04, 5 years ago

Replying to Don-vip:

@Simon: if you begin to resolve tools via Ivy, please keep error_prone, no official version actually works, a lot of patches are needed. I'm also using a Jacoco snapshot (unpatched) for Java 15 compatibility, so please wait the next release for this one. All others can be retrieved from Ivy I think.

Alright :)

comment:65 by simon04, 5 years ago

In 16166/josm:

see #16860 - Resolve SpotBugs using Apache Ivy

comment:66 by simon04, 5 years ago

In 16167/josm:

see #16860 - Separate tools/ivy.xml

comment:67 by simon04, 5 years ago

[o35377] - see #josm16860 - Resolve SpotBugs using Apache Ivy

comment:68 by simon04, 5 years ago

In 16168/josm:

see #16860 - Resolve JavaCC using Apache Ivy

comment:69 by simon04, 5 years ago

[o35378] - see #josm16860 - Resolve JavaCC using Apache Ivy

comment:70 by simon04, 5 years ago

In 16169/josm:

see #16860 - Resolve ProGuard using Apache Ivy

comment:71 by simon04, 5 years ago

In 16171/josm:

see #16860 - Resolve Checkstyle using Apache Ivy

comment:72 by simon04, 5 years ago

[o35379] - see #see16860 - Resolve Checkstyle using Apache Ivy

in reply to:  62 comment:73 by sebastic, 5 years ago

Replying to Don-vip:

@sebastic: I guess you don't need any of these test/static analysis tools for Debian packaging?

Only if they're called from build.xml via the dist target dependency chain.

comment:74 by skyper, 5 years ago

Not sure if this is the ticket to blame but there was no latest built this morning.

in reply to:  74 comment:75 by Don-vip, 5 years ago

Replying to skyper:

Not sure if this is the ticket to blame but there was no latest built this morning.

Yep, the server Makefile didn't like r16168. Looking into it

comment:76 by Don-vip, 5 years ago

Fixed. But r16165 broke ImageryCompare, looking into this now.

EDIT: fixed

Last edited 5 years ago by Don-vip (previous) (diff)

comment:77 by Don-vip, 5 years ago

In 16176/josm:

see #16860 - update Eclipse classpath

comment:78 by Don-vip, 5 years ago

@Simon: ant spotbugs is broken for plugins:

BUILD FAILED
/var/lib/jenkins/jobs/JOSM-Plugins/workspace/jdk/JDK8/build.xml:60: The following error occurred while executing this line:
/var/lib/jenkins/jobs/JOSM-Plugins/workspace/jdk/JDK8/build.xml:29: The following error occurred while executing this line:
/var/lib/jenkins/jobs/JOSM-Plugins/workspace/jdk/JDK8/build-common.xml:666: impossible to build ivy path: bad confs provided: spotbugs not found among [default]

comment:79 by simon04, 5 years ago

I cannot reproduce locally. So I'm groping in the dark. Maybe [o35389] helps.

[o35389] - see ​#josm16860 - Add ant resolve-tools target

[o35390] - see ​#josm16860 - Update to Apache Ivy 2.5.0

comment:80 by simon04, 5 years ago

I would like to get rid of source:trunk/tools/ivy/ivy.jar in the repository.

Variant 1: require Apache Ivy to be installed systemwide (as we do for Apache Ant)

Variant 2: download from https://josm.openstreetmap.de/nexus/content/repositories/public/ (as we do for plugins, as is shown in the most basic Ivy example)

  • build.xml

    diff --git a/build.xml b/build.xml
    index cc3a33e7a..11d2334c1 100644
    a b  
    1616         xmlns:unless="ant:unless"
    1717>
    1818    <target name="init-ivy">
     19        <property name="ivy.version" value="2.5.0"/>
    1920        <dirname property="base.dir" file="${ant.file.josm}"/>
    2021        <property name="lib.dir"   location="${base.dir}/lib"/>
    2122        <property name="tools.dir" location="${base.dir}/tools"/>
    2223        <property name="tools.ivy" location="${tools.dir}/ivy.xml"/>
    23         <taskdef resource="org/apache/ivy/ant/antlib.xml" uri="antlib:org.apache.ivy.ant" classpath="${tools.dir}/ivy/ivy.jar"/>
     24        <property name="ivy.jar.dir" location="${tools.dir}/ivy"/>
     25        <property name="ivy.jar.file" location="${ivy.jar.dir}/ivy-${ivy.version}.jar"/>
     26        <mkdir dir="${ivy.jar.dir}"/>
     27        <get src="https://josm.openstreetmap.de/nexus/content/repositories/public/org/apache/ivy/ivy/${ivy.version}/ivy-${ivy.version}.jar"
     28             dest="${ivy.jar.file}"
     29             skipexisting="true"
     30        />
     31        <echo>Apache Ivy ${ivy.jar.file}</echo>
     32        <taskdef resource="org/apache/ivy/ant/antlib.xml" uri="antlib:org.apache.ivy.ant" classpath="${ivy.jar.file}"/>
    2433    </target>
    2534    <target name="init-properties" depends="resolve">
    2635        <property environment="env"/>

Which variant should we take?

comment:81 by Don-vip, 5 years ago

I would say 2. Less disruptive for users. Probably as a new step that can be skipped for Debian build, or any system that provides ivy?
@sebastic your opinion?

comment:82 by sebastic, 5 years ago

No preference, the Debian package build won't call the ivy target.

in reply to:  79 comment:83 by Don-vip, 5 years ago

Replying to simon04:

I cannot reproduce locally. So I'm groping in the dark. Maybe [o35389] helps.

[o35389] - see ​#josm16860 - Add ant resolve-tools target

[o35390] - see ​#josm16860 - Update to Apache Ivy 2.5.0

It's... different :)

resolve-tools:

BUILD FAILED
/var/lib/jenkins/jobs/JOSM-Plugins/workspace/jdk/JDK8/build.xml:57: The following error occurred while executing this line:
/var/lib/jenkins/jobs/JOSM-Plugins/workspace/jdk/JDK8/build.xml:29: The following error occurred while executing this line:
/var/lib/jenkins/jobs/JOSM-Plugins/workspace/jdk/JDK8/build-common.xml:727: java.lang.NullPointerException
	at org.apache.ivy.ant.IvyAntSettings.getDefaultProperties(IvyAntSettings.java:322)
	at org.apache.ivy.ant.IvyAntSettings$1.execute(IvyAntSettings.java:268)
	at org.apache.ivy.ant.IvyAntSettings.createIvyEngine(IvyAntSettings.java:273)
	at org.apache.ivy.ant.IvyAntSettings.getDefaultInstance(IvyAntSettings.java:139)
	at org.apache.ivy.ant.IvyAntSettings.getDefaultInstance(IvyAntSettings.java:150)
	at org.apache.ivy.ant.IvyTask.getIvyInstance(IvyTask.java:77)
	at org.apache.ivy.ant.IvyTask.prepareTask(IvyTask.java:237)
	at org.apache.ivy.ant.IvyResolve.prepareTask(IvyResolve.java:236)
	at org.apache.ivy.ant.IvyTask.execute(IvyTask.java:258)
	at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:292)
	at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:99)
	at org.apache.tools.ant.Task.perform(Task.java:350)
	at org.apache.tools.ant.Target.execute(Target.java:449)

comment:84 by Don-vip, 5 years ago

The JDK11 build passed, I'm relaunching the JDK8 build.

comment:85 by simon04, 5 years ago

In 16183/josm:

see #16860 - <get> Apache Ivy, remove ivy.jar from repository

in reply to:  49 comment:86 by simon04, 5 years ago

Replying to simon04:

I've asked the apache-jcs maintainers to craft a new release which we could use via Apache Ivy: [JCS-204] Craft a new 2.3 release - https://issues.apache.org/jira/browse/JCS-204

Thomas Vandahl commented on JCS-204:

Unfortunately, the current trunk/master contains lots of breaking changes, so that a new release *must* be 3.x. This will require the package name as well as the Maven coordinates to reflect this. I'll see what I can do.

Update: I renamed the ticket to [JCS-204] Release JCS 3.0 - ASF JIRA

Last edited 5 years ago by simon04 (previous) (diff)

comment:87 by simon04, 5 years ago

@Vincent, how should we proceed with commons-jcs and jmapviewer?

Ad commons-jcs: could we publish the current SNAPSHOT to JOSM's Nexus and use 3.0-SNAPSHOT from there?

Ad jmapviewer: the latest release in JOSM's Nexus is 2.9, the latest released version 2.13 – Can we publish 2.10…2.13 to Nexus and include 2.13 via Apache Ivy

in reply to:  87 ; comment:88 by Don-vip, 5 years ago

Replying to simon04:

@Vincent, how should we proceed with commons-jcs and jmapviewer?

Ad commons-jcs: could we publish the current SNAPSHOT to JOSM's Nexus and use 3.0-SNAPSHOT from there?

Is 3.0-SNAPSHOT already available?

Ad jmapviewer: the latest release in JOSM's Nexus is 2.9, the latest released version 2.13 – Can we publish 2.10…2.13 to Nexus and include 2.13 via Apache Ivy

The jmapviewer nexus deploy currently relies on a very unreliable Jenkins job: https://josm.openstreetmap.de/jenkins/job/Nexus-JMapViewer/
We should make something most robust. Ideally from Ant? Something like an "ant release" target? For the old releases I can import them manually.

in reply to:  88 comment:89 by Don-vip, 5 years ago

Replying to Don-vip:

The jmapviewer nexus deploy currently relies on a very unreliable Jenkins job: https://josm.openstreetmap.de/jenkins/job/Nexus-JMapViewer/
We should make something most robust. Ideally from Ant? Something like an "ant release" target? For the old releases I can import them manually.

I have changed the Jenkins job to take VERSION as a parameter, seems to work. We can keep it is it until we switch to Gitlab I think. All versions should be available in Nexus now.

in reply to:  88 ; comment:90 by simon04, 5 years ago

Replying to Don-vip:

Is 3.0-SNAPSHOT already available?

3.0-SNAPSHOT is the version we are currently using via svn:externals, i.e., https://github.com/apache/commons-jcs/commit/366e4f001aaa72cf489f25d642ff88648a81a3fe
Can we compile and publish this version to our Nexus?

The jmapviewer nexus deploy currently relies on a very unreliable Jenkins job: https://josm.openstreetmap.de/jenkins/job/Nexus-JMapViewer/
We should make something most robust. Ideally from Ant? Something like an "ant release" target? For the old releases I can import them manually.

Thank you. Shall we migrate to jmapviewer 2.13 via Apache Ivy now?

Having accomplished those two steps would allow us to eliminate the svn:externals completely. :))

in reply to:  90 comment:91 by Don-vip, 5 years ago

Replying to simon04:

Can we compile and publish this version to our Nexus?

I setup an automatic build&deploy job: https://josm.openstreetmap.de/jenkins/job/Nexus-JCS/7/console
Can you please test?

Thank you. Shall we migrate to jmapviewer 2.13 via Apache Ivy now?

Sure!

Having accomplished those two steps would allow us to eliminate the svn:externals completely. :))

\o/

comment:92 by simon04, 5 years ago

In 16194/josm:

see #16860 - Resolve JMapViewer using Apache Ivy

comment:93 by simon04, 5 years ago

In 16195/josm:

see #16860 - Resolve commons-jcs-core using Apache Ivy

comment:94 by simon04, 5 years ago

Awesome, thank you! Seems to be working like a charm (at least on my system). In r16195 I had to remove the <exclude>s (i.e., not migrate them to extract-libraries) as otherwise spurious JCS/Cache exceptions would show up.

comment:95 by Don-vip, 5 years ago

Proguard fails.

comment:96 by simon04, 5 years ago

In 16197/josm:

see #16860 - Apache Ivy: exclude various commons-jcs classes from inclusion

in reply to:  95 comment:97 by simon04, 5 years ago

Replying to Don-vip:

Proguard fails.

The usual suspect ;)

comment:98 by skyper, 5 years ago

Is this ticket to blame that there was no latest released this morning ?

in reply to:  98 comment:99 by anonymous, 5 years ago

Replying to skyper:

Is this ticket to blame that there was no latest released this morning ?

no latest today, should be 16199.

comment:100 by Don-vip, 5 years ago

Ticket not to blame, random network error. Tomorrow it should work.

comment:101 by simon04, 5 years ago

In 16209/josm:

see #16860 - directory cleanup

comment:102 by GerdP, 5 years ago

After updating to r16211 I can no longer compile in Eclipse. I tried

ant clean dist

and a refresh in Eclipse but that did not help. I've also restarted Eclipse. I still have 1309 compile errors.

comment:103 by GerdP, 5 years ago

Seems all sources containing import org.openstreetmap.gui.* fail.

comment:104 by GerdP, 5 years ago

Fixed, had to do ivy resolve manually in Eclipse

in reply to:  104 ; comment:105 by Bjoeni, 5 years ago

Disclaimer: I didn't really follow / read all of this ticket and I never worked with Apache Ivy before (I only stumbled across this ticket because org.openstreetmap.gui.* failed to resolve).

But what's the best way to use JMapViewer in a JOSM plugin at the moment? I installed IvyDE in Eclipse and had it resolve, it added a reference to jmapviewer-2.13.jar. I then added that jar (from the Ivy cache directory) manually to the referenced libraries of my plugin. Is that the way to go as of right now or am I just not getting how Ivy is supposed to work?

comment:106 by Klumbumbus, 5 years ago

In 16219/josm:

see #16860, see #18880 - Adapt Netbeans config to r16165

comment:107 by Klumbumbus, 5 years ago

After r16166 Netbeans can't find spotbugs.jar anymore and the build fails, probably also due to r16194, as the message is:

C:\Users\stefa\Documents\OSM\josm\core\ide\netbeans\nbbuild.xml:90: C:\Users\stefa\Documents\OSM\josm\core\src\org\openstreetmap\gui\jmapviewer\images does not exist.
BUILD FAILED (total time: 1 minute 34 seconds)

(see also #18880)

in reply to:  105 comment:108 by taylor.smock, 5 years ago

Replying to Bjoeni:

Disclaimer: I didn't really follow / read all of this ticket and I never worked with Apache Ivy before (I only stumbled across this ticket because org.openstreetmap.gui.* failed to resolve).

But what's the best way to use JMapViewer in a JOSM plugin at the moment? I installed IvyDE in Eclipse and had it resolve, it added a reference to jmapviewer-2.13.jar. I then added that jar (from the Ivy cache directory) manually to the referenced libraries of my plugin. Is that the way to go as of right now or am I just not getting how Ivy is supposed to work?

I've had much the same issue. This (also) probably isn't the best way to go about it, but I've been changing the build path. So "Configure build path" -> "Order and Export" -> Check "Ivy". I've done this since most of my projects use JSON, and javax.json is now resolved by Ivy.

If you do that, you will have a diff for .classpath that should look like this:

  • .classpath

     
    3232                        <attribute name="test" value="true"/>
    3333                </attributes>
    3434        </classpathentry>
    35         <classpathentry kind="con" path="org.apache.ivyde.eclipse.cpcontainer.IVYDE_CONTAINER/?project=JOSM&amp;ivyXmlPath=ivy.xml&amp;confs=*&amp;ivySettingsPath=ivysettings.xml&amp;loadSettingsOnDemand=false&amp;ivyUserDir=&amp;propertyFiles="/>
     35        <classpathentry exported="true" kind="con" path="org.apache.ivyde.eclipse.cpcontainer.IVYDE_CONTAINER/?project=JOSM&amp;ivyXmlPath=ivy.xml&amp;confs=*&amp;ivySettingsPath=ivysettings.xml&amp;loadSettingsOnDemand=false&amp;ivyUserDir=&amp;propertyFiles="/>
    3636        <classpathentry kind="lib" path="test/lib/jfcunit.jar">
    3737                <attributes>
    3838                        <attribute name="test" value="true"/>

comment:109 by Don-vip, 5 years ago

In 16221/josm:

see #16860 - remove copy of jmapviewer images in Netbeans project

comment:110 by Don-vip, 5 years ago

@Klumbumbus can you please test Netbeans?

comment:111 by Klumbumbus, 5 years ago

The build works again, but it still can't find spotbugs.jar.

comment:112 by Don-vip, 5 years ago

In 16222/josm:

see #16860 - update spotbugs jar in Netbeans project

comment:113 by Don-vip, 5 years ago

Done. Please check.

comment:114 by Klumbumbus, 5 years ago

👍

comment:115 by Don-vip, 5 years ago

Resolution: fixed
Status: newclosed

comment:116 by Don-vip, 5 years ago

Resolution: fixed
Status: closedreopened

Something happened between r16155 and r16200 that broke the build for Java 14/15, but not for Java 8/11.
This is the output we get with Java 14+:

test-compile:
    [javac] Compiling 569 source files to /var/lib/jenkins/jobs/Java-EarlyAccess-JOSM/workspace/jdk/JDK14/test/build/unit
    [javac] warning: [options] bootstrap class path not set in conjunction with -source 8
    [javac] 1 warning
    [javac] Compiling 19 source files to /var/lib/jenkins/jobs/Java-EarlyAccess-JOSM/workspace/jdk/JDK14/test/build/functional
    [javac] warning: [options] bootstrap class path not set in conjunction with -source 8
    [javac] 1 warning
    [javac] Compiling 12 source files to /var/lib/jenkins/jobs/Java-EarlyAccess-JOSM/workspace/jdk/JDK14/test/build/performance
    [javac] warning: [options] bootstrap class path not set in conjunction with -source 8
    [javac] 1 warning

test:
     [echo] Running unit tests with JUnit
[jacoco:coverage] Enhancing junit with coverage
    [junit] Error: A JNI error has occurred, please check your installation and try again
    [junit] Running org.openstreetmap.josm.tools.template_engine.Batch-With-Multiple-Tests
    [junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0 sec
    [junit] Tests FAILED (crashed)
     [echo] Running functional tests with JUnit
[jacoco:coverage] Enhancing junit with coverage
    [junit] Error: A JNI error has occurred, please check your installation and try again
    [junit] Running org.openstreetmap.josm.tools.Batch-With-Multiple-Tests
    [junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0 sec
    [junit] Tests FAILED (crashed)

comment:117 by simon04, 5 years ago

I cannot reproduce locally with r16225

$ gradle --version
...
Ant:          Apache Ant(TM) version 1.10.7 compiled on September 1 2019
JVM:          14 (AdoptOpenJDK 14+36)

$ ant clean

$ ant test '-Ddefault-junit-includes=**/josm/tools/**/*Test.class'
...
test:
     [echo] Running unit tests with JUnit
...
BUILD SUCCESSFUL
Total time: 1 minute 11 seconds

in reply to:  60 ; comment:118 by simon04, 5 years ago

Replying to Don-vip:

@Simon: if you begin to resolve tools via Ivy, please keep error_prone, no official version actually works, a lot of patches are needed. I'm also using a Jacoco snapshot (unpatched) for Java 15 compatibility, so please wait the next release for this one.

Can we deploy those snapshots to https://josm.openstreetmap.de/nexus/content/repositories/snapshots/ as we've done for Apache Commons JCS? Then we could also resolve all test dependencies using Apache Ivy.

JaCoCo got official Java 14 support in the most recent commit (with the nice commit message "Happy birthday JDK 14!"): https://github.com/jacoco/jacoco/commit/1c5601bba5bf195a5f9b09a87cb5d2152857bbb7#diff-0886964cfc7b181768d1d9b182f50528R25

comment:119 by Don-vip, 5 years ago

Ah you're right it's just that I enabled Jacoco coverage in r16156. Strangely it works also for me locally, even in headless mode.

in reply to:  118 ; comment:120 by Don-vip, 5 years ago

Replying to simon04:

Can we deploy those snapshots to https://josm.openstreetmap.de/nexus/content/repositories/snapshots/ as we've done for Apache Commons JCS?

No need, they have their own maven repo for snapshots.

Then we could also resolve all test dependencies using Apache Ivy.

I'm trying to do it :)

in reply to:  120 comment:121 by Don-vip, 5 years ago

Replying to Don-vip:

I'm trying to do it :)

I have bad luck: the very moment I try to use their snapshots repository, it goes down: https://status.maven.org/incidents/bwklmrfj9z1c

in reply to:  120 comment:122 by simon04, 5 years ago

Replying to Don-vip:

No need, they have their own maven repo for snapshots.

Convenient :-)

Replying to Don-vip:

I have bad luck: the very moment I try to use their snapshots repository, it goes down

:-D
It's up again:

Quoting https://status.maven.org/incidents/bwklmrfj9z1c

Services have been restored on oss.sonatype.org.
Posted 11 minutes ago. Apr 04, 2020 - 09:20 EDT

comment:123 by Don-vip, 5 years ago

In 16229/josm:

see #16860 - resolve test libraries using Ivy

comment:124 by Don-vip, 5 years ago

In 16231/josm:

see #16860 - fix compilation of scripts

comment:125 by Don-vip, 5 years ago

Priority: normalmajor

comment:126 by Don-vip, 5 years ago

Resolution: fixed
Status: reopenedclosed

Found out what the Java 14 issue was. It came from the Jenkins job configuration:

jacoco.includes=org.openstreetmap.josm.*:java.*:javax.*:com.sun.*:sun.*
jacoco.inclbootstrapclasses=true
jacoco.inclnolocationclasses=true

I think I tried a long time ago to cover JDK classes too, when we were part of the OpenJDK Outreach, before Oracle decided to go full $$$ on Java.
I removed this configuration and now it works :)

comment:127 by Don-vip, 5 years ago

Plugin build updated in [o35411:35412].

Last edited 5 years ago by Don-vip (previous) (diff)

comment:128 by simon04, 5 years ago

[o35413] - resolve test libraries using Ivy settings from core

comment:129 by GerdP, 5 years ago

Is there an easy way to suppress all the [ivy:resolve] messages?
When I use e.g.

ant checkstyle pmd spotbugs

it is difficult to find the messages from checkstyle and checkstyle-josm.xml is hard to read because of its length.
I also wonder why I see those for each target. Is that intended?
For now I try to get used to a different order:

ant pmd spotbugs checkstyle 

comment:130 by simon04, 5 years ago

In 16243/josm:

see #16860 - Apache Ivy: tune logging

comment:131 by GerdP, 5 years ago

thanks!

in reply to:  41 comment:132 by sebastic, 5 years ago

Replying to simon04:

The josm-sources.jar would contain all JOSM's own sources (.java files + resources) and all of its dependencies (plenty of additional .java files + resources)? This seems doable, but requires some investigations into Ant/Ivy (at least, I don't know how to accomplish this by heart).

The current source JAR is not sufficient, there are some transitive dependencies, to build the subset of metadata-extractor required for JOSM the adobe-xmp-core sources are also required, but it's license terms are non-free i.e. incompatible with the DFSG & OSD.

The OpeningHoursParser sources also cannot be built because the ParseException & Token symbols cannot be found. This seems to be missing imports. It may also be related to using JDK 11 to build JOSM whereas OpeningHoursParser likely still uses JDK 8.

Update: Compiling the metadata-extractor sources was possible after adding the sources from xmpcore-6.0.6-sources.jar (BSD-3-Clause), and compiling the OpeningHoursParser sources after adding the sources from annotations-19.0.0-sources.jar (Apache-2.0) and a few generated sources.

Last edited 5 years ago by sebastic (previous) (diff)

comment:133 by simon04, 5 years ago

In 16338/josm:

see #16860 - Apache Ivy: update xmpcore to fix warning

comment:134 by simon04, 5 years ago

In 16339/josm:

see #16860 - Apache Ivy: remove unused epsg configuration

comment:135 by GerdP, 5 years ago

Please check https://josm.openstreetmap.de/jenkins/job/JOSM-Plugins/jdk=JDK8/1842/console

BUILD FAILED
/var/lib/jenkins/jobs/JOSM-Plugins/workspace/jdk/JDK8/build.xml:54: The following error occurred while executing this line:
/var/lib/jenkins/jobs/JOSM-Plugins/workspace/jdk/JDK8/build.xml:29: The following error occurred while executing this line:
/var/lib/jenkins/jobs/JOSM-Plugins/workspace/jdk/JDK8/build-common.xml:551: impossible to ivy retrieve: java.lang.RuntimeException: problem during retrieve of org.openstreetmap.josm.plugins#wikipedia: java.lang.IllegalStateException: Report file '/var/lib/jenkins/.ant/cache/org.openstreetmap.josm.plugins-wikipedia-test.xml' does not exist.

comment:136 by stoecker, 5 years ago

Build in plugins fails:

~/josm/plugins/apache-commons> ant
Buildfile: /home/stoecker/sources/svn.openstreetmap.org/applications/editors/josm/plugins/apache-commons/build.xml

resolve:

BUILD FAILED
/home/stoecker/sources/svn.openstreetmap.org/applications/editors/josm/plugins/apache-commons/build.xml:23: Problem: failed to create task or type antlib:org.apache.ivy.ant:retrieve
Cause: The name is undefined.
Action: Check the spelling.
Action: Check that any custom tasks/types have been declared.
Action: Check that any <presetdef>/<macrodef> declarations have taken place.
No types or tasks have been defined in this namespace yet

This appears to be an antlib declaration. 
Action: Check that the implementing library exists in one of:
        -/usr/share/ant/lib
        -/home/stoecker/.ant/lib
        -a directory added on the command line with the -lib argument


Total time: 0 seconds

comment:137 by stoecker, 5 years ago

Resolution: fixed
Status: closedreopened

comment:138 by simon04, 5 years ago

In 16403/josm:

see #16860 - Resolve oauth.signpost:signpost-core via Apache Ivy

Signpost 2.0.0 includes the update to Java 8: https://github.com/mttkay/signpost/releases/tag/oauth-signpost-2.0.0

comment:139 by GerdP, 5 years ago

plugin pdfimport is missing org.apache.commons.logging and others:
https://josm.openstreetmap.de/jenkins/job/JOSM-Plugins/jdk=JDK8/1850/consoleText
I assume something reg. ivy is missing?

comment:140 by GerdP, 5 years ago

I can compile the plugin when I uncomment the line in the build.xml

<!-- <property name="plugin.requires" value="apache-commons"/> -->

Why is this line needed again now? It was commented in 2018 and the build worked in April 2020

comment:141 by taylor.smock, 5 years ago

Possibly. It looks like there used to be a src/org/apache/commons/logging directory in the source code, which is why the plugin was compiling correctly (the logging library removed in r16050).

Do you know if there was a reason for using a logging library directly instead of JOSM Logging?

comment:142 by GerdP, 5 years ago

I don't understand what's going on. I've reverted my changes and now ant clean dist works in directory pdfimport.

comment:143 by GerdP, 5 years ago

.. and now its back again. Seems a dependency is missing?

comment:144 by simon04, 5 years ago

commons-logging via Ivy: r16024
commons-logging dropped as it is nowhere used in JOSM core: r16397

comment:145 by GerdP, 5 years ago

So, what should be done with plugins? Is

<property name="plugin.requires" value="apache-commons"/> 

the solution?

comment:146 by taylor.smock, 5 years ago

It is probably the quickest solution.

But all the pdfimport plugin is doing is creating a log instance (with Apache Logging LogFactory), and then logging exceptions to it (log.warn). I think that pdfimport can, at least, get by without using the Apache Logging library. I think that the loggers could be replaced with Logging.warn(e) with no loss of functionality.

comment:147 by simon04, 5 years ago

pdfimport: removed commong.logging usages in [o35448:35449].

However, pdfbox internally uses log4j according to https://mvnrepository.com/artifact/org.apache.pdfbox/pdfbox/1.8.12 and the following warning is printed (I can live with that):

log4j:WARN No appenders could be found for logger (org.apache.pdfbox.pdfparser.PDFObjectStreamParser).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

comment:148 by simon04, 5 years ago

In 16408/josm:

see #16860 - build.xml: remove leftovers of org.apache.commons.logging

comment:149 by Don-vip, 4 years ago

In 16536/josm:

see #16860 - remove leftovers of OAuth signpost sources

comment:150 by Don-vip, 4 years ago

Can't we remove com/google/gdata/util/common/base? According to r4231 it seems used by signpost and/or metadata extractor, both are managed by Ivy now.

comment:151 by Don-vip, 4 years ago

Milestone: 20.0320.05

comment:152 by simon04, 4 years ago

In 16537/josm:

see #16860 - Remove src/com/google/gdata

Leftover from r16403

comment:153 by Don-vip, 4 years ago

Only one dependency to go \o/

comment:154 by simon04, 4 years ago

The hardest one … Might not be ready for the 20.05 milestone … ;-)

Should we release r16538 as 20.05?

in reply to:  154 ; comment:155 by Don-vip, 4 years ago

Milestone: 20.0520.06

Replying to simon04:

The hardest one … Might not be ready for the 20.05 milestone … ;-)

Should we release r16538 as 20.05?

If you feel it's stable enough, let's do it. I didn't follow JOSM development on last month.

in reply to:  155 comment:156 by stoecker, 4 years ago

Replying to Don-vip:

Replying to simon04:

The hardest one … Might not be ready for the 20.05 milestone … ;-)

Should we release r16538 as 20.05?

If you feel it's stable enough, let's do it. I didn't follow JOSM development on last month.

You have to wait till tomorrow, as this version lies behind the nightly build. I've not seen anything major in the new tickets lately, so I also have no objections. I actually wanted to set sundays version but there were new checkins all the time.

comment:157 by simon04, 4 years ago

  • r16536 is the built latest version.
  • r16537 is irrelevant for the release.
  • r16538 is an i18n update of a few languages.

If we can trigger a latest build for r16538, let's take this one. Otherwise r16536 is good. :-)

Thanks!

comment:158 by stoecker, 4 years ago

We waited so long, we can wait another day :-)

comment:159 by Don-vip, 4 years ago

I just launched a manual build :)

comment:160 by simon04, 4 years ago

In 16603/josm:

see #16860 - Suppress "bad path element .../xmpcore-6.0.6.jar: no such file or directory" warning

comment:161 by simon04, 4 years ago

Milestone: 20.06Longterm

comment:162 by simon04, 4 years ago

In 17128/josm:

see #16860 - Apache Ivy: only use josm-nexus repository by default

comment:163 by GerdP, 4 years ago

What are the required steps to configure ivy so that it doesn't print lots of these error messages when compiling a plugin like e.g.. reverter?
[ivy:resolve] unknown resolver josm-nexus

comment:164 by GerdP, 4 years ago

In the wikipedia plugin directory I find this ivy_settings.xml :

<?xml version="1.0" encoding="utf-8"?>
<ivysettings>
  <settings defaultResolver="josm-nexus"/>
  <resolvers>
    <ibiblio name="josm-nexus" m2compatible="true" root="https://josm.openstreetmap.de/nexus/content/repositories/public/" />
  </resolvers>
</ivysettings>

When I copy this file into the reverter directory the messages disappear. Does that mean we need this file in every plugin directory that doesn't yet have one? Or is it possible to have a working default for all of them?

comment:165 by Don-vip, 4 years ago

In 35657/osm:

see #16860 - upgrade to JNA 5.6.0 + switch to Ivy

comment:166 by simon04, 3 years ago

In 35764/osm:

see #16860 - ivy_settings: consistently use "josm-nexus"

Try to solve "unknown resolver josm-nexus" warnings in Jenkins.

comment:167 by simon04, 3 years ago

In 35766/osm:

see #16860 - Apache Ivy: fallback to core ivysettings.xml when plugin ivy_settings.xml does not exist

comment:168 by simon04, 3 years ago

An attempt to merge ivy.xml and tools/ivy.xml – https://github.com/openstreetmap/josm/compare/one-ivy

Rationale: one file for all dependencies. javacc and errorprone are currently listed in tools/ivy.xml, but are needed for compiling JOSM.

Modify Ticket

Change Properties
Set your email in Preferences
Action
as reopened The owner will remain team.
as The resolution will be set. Next status will be 'closed'.
to The owner will be changed from team to the specified user. Next status will be 'new'.
Next status will be 'needinfo'. The owner will be changed from team to wiktorn.
as duplicate The resolution will be set to duplicate. Next status will be 'closed'. The specified ticket will be cross-referenced with this ticket.

Add Comment


E-mail address and name can be saved in the Preferences .
 
Note: See TracTickets for help on using tickets.