Modify

Opened 2 years ago

Last modified 5 weeks ago

#16860 reopened enhancement

Resolve all dependencies and tools using Apache Ivy

Reported by: wiktorn Owned by: team
Priority: major Milestone: Longterm
Component: Core Version:
Keywords: hack-weekend-2018-10 Cc: stoecker, sebastic

Description (last modified by wiktorn)

Follow up to: #8269 and #16420. Needed for: #16858

Use JMapViewer from our Nexus instead of use of svn:externals. This allows us to:

  • test in the field, how well does it's job in controlling our dependencies
  • move JMapViewer outside OSM SVN server

The goal is to do incremental changes and provide better dependency management than following HEAD on svn:externals.

Sponsored by: OpenStreetMap Hack Weekend, Karlsruhe Oct'18

Attachments (2)

ivy-jmapviewer.patch (13.8 KB) - added by wiktorn 2 years ago.
ivy-jmapviewer-v2.patch (12.3 KB) - added by simon04 2 years ago.

Download all attachments as: .zip

Change History (166)

Changed 2 years ago by wiktorn

Attachment: ivy-jmapviewer.patch added

comment:1 Changed 2 years ago by wiktorn

Description: modified (diff)
Keywords: hack-weekend-2018-10 added

comment:2 Changed 2 years ago by Don-vip

Milestone: 18.1018.11

comment:3 Changed 2 years ago by Don-vip

Milestone: 18.1118.12

comment:4 Changed 2 years ago by Don-vip

Milestone: 18.1219.01

Changed 2 years ago by simon04

Attachment: ivy-jmapviewer-v2.patch added

comment:5 Changed 2 years ago by simon04

I merged the patch with the recent changes and uploaded attachment:ivy-jmapviewer-v2.patch

Is there anything holding us back?

comment:6 in reply to:  5 Changed 2 years ago by stoecker

Is there anything holding us back?

Yes. The new server. We don't want to install and do anything new on the old one.

comment:7 Changed 23 months ago by Don-vip

Milestone: 19.0119.02

comment:8 Changed 22 months ago by Don-vip

Milestone: 19.0219.03

comment:9 Changed 20 months ago by Don-vip

Milestone: 19.0319.04

comment:10 Changed 20 months ago by Don-vip

Milestone: 19.0419.05

comment:11 Changed 19 months ago by Don-vip

Milestone: 19.05

comment:12 Changed 10 months ago by simon04

Milestone: 20.02

Since we got a new server in the meanwhile, is anything (else) holding us back?

comment:13 in reply to:  12 Changed 10 months ago by Don-vip

Replying to simon04:

is anything (else) holding us back?

I don't know yet how to complete the GitLab setup. I'll comment why in #16857 comments.

comment:14 Changed 10 months ago by simon04

Milestone: 20.0220.03

comment:15 Changed 9 months ago by simon04

In 15977/josm:

see #16860, see #18140 - Setup Apache Ivy (patch by wiktorn, modified)

comment:16 Changed 9 months ago by simon04

In 16011/josm:

see #16860, see #18140 - Restrict SpotBugs to org/openstreetmap/josm

comment:17 Changed 9 months ago by Don-vip

In 16020/josm:

see #16860, see #18140 - better way to restrict SpotBugs analysis scope (without error about missing classes)

See https://github.com/spotbugs/spotbugs/blob/4.0.0/spotbugs-ant/src/main/java/edu/umd/cs/findbugs/anttask/FindBugsTask.java

comment:18 Changed 9 months ago by simon04

In 16021/josm:

see #16860 - Resolve org.glassfish:javax.json via Apache Ivy

comment:19 Changed 9 months ago by simon04

In 16022/josm:

see #16860 - Resolve org.tukaani:xz via Apache Ivy

comment:20 Changed 9 months ago by simon04

In 16023/josm:

see #16860 - Resolve org.apache.commons:commons-compress via Apache Ivy

comment:21 Changed 9 months ago by simon04

In 16024/josm:

see #16860 - Resolve commons-logging:commons-logging via Apache Ivy

comment:22 Changed 9 months ago by simon04

In 16025/josm:

see #16860 - Resolve com.drewnoakes:metadata-extractor via Apache Ivy

comment:23 in reply to:  22 ; Changed 9 months ago by Don-vip

Replying to simon04:

In 16025/josm:

see #16860 - Resolve com.drewnoakes:metadata-extractor via Apache Ivy

Are you sure it works for this one? I remember I must patch it at each update.

comment:24 Changed 9 months ago by simon04

Here's how the dist sizes evolved:

ls -l --time-style full-iso 
total 84764
-rw-r--r-- 1 simon users 14443614 2020-03-03 22:21:13 +0100 josm-custom-16016.jar
-rw-r--r-- 1 simon users 14444368 2020-03-03 22:23:10 +0100 josm-custom-16016+json.jar
-rw-r--r-- 1 simon users 14429230 2020-03-03 22:24:45 +0100 josm-custom-16016+json+xz.jar
-rw-r--r-- 1 simon users 14432267 2020-03-03 22:37:35 +0100 josm-custom-16016+json+xz+compress.jar
-rw-r--r-- 1 simon users 14430688 2020-03-03 22:40:41 +0100 josm-custom-16016+json+xz+compress+logging.jar
-rw-r--r-- 1 simon users 14604031 2020-03-03 23:28:36 +0100 josm-custom-16016+json+xz+compress+logging+metadata-extractor.jar

The increase for metadata-extractor is due to our disabled imports in JpegMetadataReader.java which cannot be ported to Apache Ivy.


How to proceed?

comment:25 in reply to:  23 ; Changed 9 months ago by simon04

Replying to Don-vip:

Are you sure it works for this one? I remember I must patch it at each update.

Haven't found any problem yet. I successfully opened a bunch of JPG images; ExifReaderTest passes.

comment:26 in reply to:  24 Changed 9 months ago by Don-vip

Replying to simon04:

The increase for metadata-extractor is due to our disabled imports in JpegMetadataReader.java which cannot be ported to Apache Ivy.

We could ask the author to provide smaller artifacts.

comment:27 in reply to:  24 ; Changed 9 months ago by Don-vip

Replying to simon04:

  • signpost: we did some minor cleanups and kicked out commons-codecs in r8149, r10746

Signpost is dead, no commit in 5 years, 15 open pull requests...

Did we consider https://github.com/scribejava/scribejava at some point? It looks way more active, but probably heavier.

comment:28 in reply to:  25 ; Changed 9 months ago by Don-vip

Replying to simon04:

Replying to Don-vip:

Are you sure it works for this one? I remember I must patch it at each update.

Haven't found any problem yet. I successfully opened a bunch of JPG images; ExifReaderTest passes.

I'm almost sure it doesn't work. Look at proguard output:

 [proguard] Warning: com.drew.imaging.FileTypeDetector: can't find referenced class com.drew.imaging.quicktime.QuickTimeTypeChecker
 [proguard] Warning: com.drew.imaging.FileTypeDetector: can't find referenced class com.drew.imaging.quicktime.QuickTimeTypeChecker
 [proguard] Warning: com.drew.imaging.FileTypeDetector: can't find referenced class com.drew.imaging.riff.RiffTypeChecker
 [proguard] Warning: com.drew.imaging.FileTypeDetector: can't find referenced class com.drew.imaging.riff.RiffTypeChecker
 [proguard] Warning: com.drew.imaging.FileTypeDetector: can't find referenced class com.drew.imaging.mp3.MpegAudioTypeChecker
 [proguard] Warning: com.drew.imaging.FileTypeDetector: can't find referenced class com.drew.imaging.mp3.MpegAudioTypeChecker
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.psd.PsdMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.png.PngMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.bmp.BmpMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.gif.GifMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.ico.IcoMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.pcx.PcxMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.webp.WebpMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.raf.RafMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.avi.AviMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.wav.WavMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.quicktime.QuickTimeMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.mp4.Mp4MetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.mp3.Mp3MetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.eps.EpsMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.heif.HeifMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.psd.PsdMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.png.PngMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.bmp.BmpMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.gif.GifMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.ico.IcoMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.pcx.PcxMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.webp.WebpMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.raf.RafMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.avi.AviMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.wav.WavMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.quicktime.QuickTimeMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.mp4.Mp4MetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.mp3.Mp3MetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.eps.EpsMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.heif.HeifMetadataReader
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$1: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamFactory
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$1: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamFactory
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$1: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamFactory
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$1: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamFactory
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$1: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamProvider
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$1: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamProvider
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$1: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamFactory
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$1: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamProvider
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$1: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamProvider
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$1: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamProvider
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$1: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamProvider
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$1: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamFactory
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$2: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamFactory
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$2: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamFactory
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$2: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamFactory
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$2: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamFactory
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$2: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamProvider
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$2: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamProvider
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$2: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamFactory
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$2: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamProvider
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$2: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamProvider
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$2: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamProvider
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$2: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamProvider
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$2: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamFactory
 [proguard] Warning: org.apache.commons.compress.compressors.xz.XZUtils$CachedAvailability: can't find referenced class org.apache.commons.compress.compressors.xz.XZUtils
 [proguard] Warning: there were 61 unresolved references to classes or interfaces.

comment:29 in reply to:  27 Changed 9 months ago by simon04

Replying to Don-vip:

Did we consider https://github.com/scribejava/scribejava at some point? It looks way more active, but probably heavier.

Half an hour ago, I've also found this library. Looks interesting. Will investigate in the next days. The first thing I noticed is that com.github.scribejava.core.extractors.AbstractOAuth1JSONTokenExtractor makes use of Jackson for JSON decoding, but we could probably provide our own implementation.

comment:30 in reply to:  28 Changed 9 months ago by simon04

Replying to Don-vip:

I'm almost sure it doesn't work. Look at proguard output:

This is the list of classes which are now included in josm.jar, but haven't been before:

$ comm -3 all-classes-in-josm.jar-before all-classes-in-josm.jar-after
        com/adobe/
        com/adobe/internal/
        com/adobe/internal/xmp/
        com/adobe/internal/xmp/impl/
        com/adobe/internal/xmp/impl/Base64.class
        com/adobe/internal/xmp/impl/ByteBuffer.class
        com/adobe/internal/xmp/impl/CountOutputStream.class
        com/adobe/internal/xmp/impl/FixASCIIControlsReader.class
        com/adobe/internal/xmp/impl/ISO8601Converter.class
        com/adobe/internal/xmp/impl/Latin1Converter.class
        com/adobe/internal/xmp/impl/ParameterAsserts.class
        com/adobe/internal/xmp/impl/ParseRDF.class
        com/adobe/internal/xmp/impl/ParseState.class
        com/adobe/internal/xmp/impl/QName.class
        com/adobe/internal/xmp/impl/Utils.class
        com/adobe/internal/xmp/impl/XMPDateTimeImpl.class
        com/adobe/internal/xmp/impl/XMPIteratorImpl$NodeIterator$1.class
        com/adobe/internal/xmp/impl/XMPIteratorImpl$NodeIteratorChildren.class
        com/adobe/internal/xmp/impl/XMPIteratorImpl$NodeIterator.class
        com/adobe/internal/xmp/impl/XMPIteratorImpl.class
        com/adobe/internal/xmp/impl/XMPMetaImpl$1.class
        com/adobe/internal/xmp/impl/XMPMetaImpl$2.class
        com/adobe/internal/xmp/impl/XMPMetaImpl.class
        com/adobe/internal/xmp/impl/XMPMetaParser.class
        com/adobe/internal/xmp/impl/XMPNode$1.class
        com/adobe/internal/xmp/impl/XMPNode.class
        com/adobe/internal/xmp/impl/XMPNodeUtils.class
        com/adobe/internal/xmp/impl/XMPNormalizer.class
        com/adobe/internal/xmp/impl/XMPSchemaRegistryImpl$1.class
        com/adobe/internal/xmp/impl/XMPSchemaRegistryImpl.class
        com/adobe/internal/xmp/impl/XMPSerializerHelper.class
        com/adobe/internal/xmp/impl/XMPSerializerRDF.class
        com/adobe/internal/xmp/impl/XMPUtilsImpl.class
        com/adobe/internal/xmp/impl/xpath/
        com/adobe/internal/xmp/impl/xpath/PathPosition.class
        com/adobe/internal/xmp/impl/xpath/XMPPath.class
        com/adobe/internal/xmp/impl/xpath/XMPPathParser.class
        com/adobe/internal/xmp/impl/xpath/XMPPathSegment.class
        com/adobe/internal/xmp/options/
        com/adobe/internal/xmp/options/AliasOptions.class
        com/adobe/internal/xmp/options/IteratorOptions.class
        com/adobe/internal/xmp/options/Options.class
        com/adobe/internal/xmp/options/ParseOptions.class
        com/adobe/internal/xmp/options/PropertyOptions.class
        com/adobe/internal/xmp/options/SerializeOptions.class
        com/adobe/internal/xmp/properties/
        com/adobe/internal/xmp/properties/XMPAliasInfo.class
        com/adobe/internal/xmp/properties/XMPProperty.class
        com/adobe/internal/xmp/properties/XMPPropertyInfo.class
        com/adobe/internal/xmp/utils/
        com/adobe/internal/xmp/utils/XMLStreamWriterFactory.class
        com/adobe/internal/xmp/utils/XMLStreamWriterImpl.class
        com/adobe/internal/xmp/XMPConst.class
        com/adobe/internal/xmp/XMPDateTime.class
        com/adobe/internal/xmp/XMPDateTimeFactory.class
        com/adobe/internal/xmp/XMPError.class
        com/adobe/internal/xmp/XMPException.class
        com/adobe/internal/xmp/XMPIterator.class
        com/adobe/internal/xmp/XMPMeta.class
        com/adobe/internal/xmp/XMPMetaFactory$1.class
        com/adobe/internal/xmp/XMPMetaFactory.class
        com/adobe/internal/xmp/XMPPathFactory.class
        com/adobe/internal/xmp/XMPSchemaRegistry.class
        com/adobe/internal/xmp/XMPUtils.class
        com/adobe/internal/xmp/XMPVersionInfo.class
        com/drew/imaging/FileType.class
        com/drew/imaging/FileTypeDetector.class
        com/drew/imaging/ImageMetadataReader$1.class
        com/drew/imaging/ImageMetadataReader.class
        com/drew/imaging/tiff/TiffMetadataReader.class
        com/drew/imaging/TypeChecker.class
        com/drew/lang/ByteConvert.class
        com/drew/lang/ByteTrie$ByteTrieNode.class
        com/drew/lang/ByteTrie.class
        com/drew/lang/ByteUtil.class
        com/drew/lang/Iterables.class
        com/drew/lang/KeyValuePair.class
        com/drew/lang/NullOutputStream.class
        com/drew/lang/RandomAccessFileReader.class
        com/drew/lang/RandomAccessStreamReader.class
        com/drew/lang/StreamUtil.class
        com/drew/metadata/adobe/
        com/drew/metadata/adobe/AdobeJpegDescriptor.class
        com/drew/metadata/adobe/AdobeJpegDirectory.class
        com/drew/metadata/adobe/AdobeJpegReader.class
        com/drew/metadata/file/FileTypeDescriptor.class
        com/drew/metadata/file/FileTypeDirectory.class
        com/drew/metadata/jfif/
        com/drew/metadata/jfif/JfifDescriptor.class
        com/drew/metadata/jfif/JfifDirectory.class
        com/drew/metadata/jfif/JfifReader.class
        com/drew/metadata/jfxx/
        com/drew/metadata/jfxx/JfxxDescriptor.class
        com/drew/metadata/jfxx/JfxxDirectory.class
        com/drew/metadata/jfxx/JfxxReader.class
        com/drew/metadata/Schema.class
        com/drew/metadata/xmp/
        com/drew/metadata/xmp/XmpDescriptor.class
        com/drew/metadata/xmp/XmpDirectory.class
        com/drew/metadata/xmp/XmpReader.class
        com/drew/metadata/xmp/XmpWriter.class

In particular, FileTypeDetector and ImageMetadataReader are present in this list. We could exclude all those …

Last edited 9 months ago by simon04 (previous) (diff)

comment:31 Changed 9 months ago by simon04

In 16026/josm:

see #16860 - Apache Ivy: exclude various subclasses from inclusion

comment:32 Changed 9 months ago by simon04

In 16027/josm:

see #16860 - Apache Ivy: exclude com/drew/imaging/{FileTypeDetector,ImageMetadataReader} from inclusion

comment:33 Changed 9 months ago by Don-vip

In 16050/josm:

see #18845 - see #16860 - directory cleanup

comment:34 in reply to:  27 Changed 9 months ago by simon04

Replying to Don-vip:

Replying to simon04:

  • signpost: we did some minor cleanups and kicked out commons-codecs in r8149, r10746

Signpost is dead, no commit in 5 years, 15 open pull requests...

I posted to the signpost mailing list: https://groups.google.com/forum/#!msg/signpost-users/QDccoNNTvpc/5pRpdLZyAAAJ

comment:35 Changed 9 months ago by simon04

Cc: sebastic added

Hi sebastic, this might affect the packaging for Debian. A few libraries are obtained as JAR via Apache Ivy, see source:trunk/ivy.xml

comment:36 Changed 9 months ago by sebastic

move JMapViewer outside OSM SVN server

Where is it going? There is no JMapViewer repo on GitHub yet.

The Debian package uses the ZIP releases from https://svn.openstreetmap.org/applications/viewer/jmapviewer/releases/.

In the ivy config I see metadata-extractor, we used to build the josm package with the separately packaged metadata-extractor but because its API is highly unstable updating always broke either josm or one of the other dependent packages.

If the sources for unstable dependencies like this won't be bundled with josm any more, we won't be able to keep to the josm package in Debian.

comment:37 Changed 9 months ago by simon04

The goal is to use the official JAR distributions for external dependencie, and not (partially) compile them within the JOSM build system.

The current status:

  • Apache Commons Compress ✔
  • Apache Commons JCS: TODO, needs new release, see comment:24
  • Apache Commons Logging ✔
  • SvgSalamander: UNLIKELY
  • Metadata Extractor ✔
  • Signpost: TODO, WIP, see comment:34
  • Jsonp ✔
  • OpeningHoursValidator ✔
  • JMapViewer: TODO (needs separate repository first)

Running ant dist depends on ant resolve (which runs Apache Ivy to fetch the dependencies) and bundles the dependencies in a fatjar. So, by default all dependencies are bundled into dist/josm-custom.jar.

comment:38 Changed 9 months ago by sebastic

ant resolve doesn't work in a Debian build environment as it doesn't have network, the dependencies need to have their source included in the source tree or an appropriate version needs to be packaged and installed in the build environment.

Experience has shown that using separately packaged metadata-extractor and svgsalamander made backporting JOSM tested for Debian stable a PITA. Once the embedded copies were used this was much less of a problem. The separately packaged JMapViewer has also broken freeplane on more than one occasion, forcing stable users to choose whether to have the josm backport installed (which requires a newer jmapviewer) or a working freeplane.

Because these recent changes make building JOSM from source in a Debian build environment unfeasible when backports are required as well, we'll have to remove the josm package from Debian and replace it with an installer that downloads the pre-built JAR. This is not very good either (because the JARs don't have hashes nor cryptographic signatures to verify the download with), but at least it will keep JOSM tested easily available to users of Debian stable (assuming the josm-installer package passes FTP master review).

comment:39 Changed 9 months ago by simon04

Is there a Debian compatible way not involving svn:externals? By no means, the goal was to make Debian packging harder/impossible. However, slowly getting rid of svn:externals allows us to move forward in the long-term goal #16871.

comment:40 Changed 9 months ago by sebastic

Publish a source jar that bundles the required versions of the dependencies?

comment:41 Changed 9 months ago by simon04

The josm-sources.jar would contain all JOSM's own sources (.java files + resources) and all of its dependencies (plenty of additional .java files + resources)? This seems doable, but requires some investigations into Ant/Ivy (at least, I don't know how to accomplish this by heart).

comment:42 in reply to:  38 ; Changed 9 months ago by wiktorn

Replying to sebastic:

Because these recent changes make building JOSM from source in a Debian build environment unfeasible when backports are required as well, we'll have to remove the josm package from Debian and replace it with an installer that downloads the pre-built JAR. This is not very good either (because the JARs don't have hashes nor cryptographic signatures to verify the download with), but at least it will keep JOSM tested easily available to users of Debian stable (assuming the josm-installer package passes FTP master review).

Jars published on JOSM page are signed and it is possibile to verify the signature. See: https://docs.oracle.com/en/java/javase/11/tools/jarsigner.html

Replying to simon04:

The josm-sources.jar would contain all JOSM's own sources (.java files + resources) and all of its dependencies (plenty of additional .java files + resources)? This seems doable, but requires some investigations into Ant/Ivy (at least, I don't know how to accomplish this by heart).

If I remember correctly, it was possibile to download sources in Ivy

comment:43 Changed 9 months ago by Don-vip

Another Ivy issue we need to fix: proxy support, see https://lists.openstreetmap.org/pipermail/josm-dev/2020-March/008288.html

comment:44 in reply to:  42 Changed 9 months ago by sebastic

Replying to wiktorn:

Jars published on JOSM page are signed and it is possibile to verify the signature. See: https://docs.oracle.com/en/java/javase/11/tools/jarsigner.html

That's not a equivalent to detached GPG signatures.

jarsigner is part of the JDK, installing the entire JDK to run JOSM is highly undesirable.

comment:45 Changed 9 months ago by Don-vip

In 16118/josm:

see #16860 - instruct Eclipse IvyDE plugin to use ivysettings.xml

comment:46 Changed 9 months ago by Don-vip

In 16125/josm:

see #16860 - see #18880 - explicit use of Ivy configuration files for Netbeans compatibility

comment:47 Changed 9 months ago by Don-vip

In 16139/josm:

see #16860 - see #18880 - setup Ivy in Netbeans project

comment:48 Changed 9 months ago by simon04

In 16141/josm:

see #16860 - build.xml: add ant sources

Generates jar file of JOSM source files and its dependencies.

comment:49 Changed 9 months ago by simon04

I've asked the apache-jcs maintainers to craft a new release which we could use via Apache Ivy: [JCS-204] Craft a new 2.3 release - https://issues.apache.org/jira/browse/JCS-204

comment:50 in reply to:  48 ; Changed 9 months ago by Don-vip

Replying to simon04:

In 16141/josm:

see #16860 - build.xml: add ant sources

Generates jar file of JOSM source files and its dependencies.

@sebastic: I guess you don't need the REVISION file?

@simon: Dropping the "create-revision" dependency would make it easier for me to include this new task in the server Makefile

comment:51 in reply to:  50 Changed 9 months ago by sebastic

Replying to Don-vip:

@sebastic: I guess you don't need the REVISION file?

The package already creates it itself:

https://salsa.debian.org/debian-gis-team/josm/-/blob/debian/0.0.svn15937+dfsg-1/debian/rules#L80

And it's patched to include Build-Name & Debian-Release:

https://salsa.debian.org/debian-gis-team/josm/-/blob/debian/0.0.svn15937+dfsg-1/debian/patches/00-build.patch#L39

comment:52 Changed 9 months ago by simon04

In 16150/josm:

see #16860 - build.xml: ant sources: remove REVISION

comment:53 Changed 9 months ago by Don-vip

@sebastic can you please check if https://josm.openstreetmap.de/download/josm-latest-sources.jar is enough for you to build a JOSM Debian package.

comment:54 Changed 9 months ago by sebastic

JOSM tested 15937 contained these subdirectories under src/ which are not included in the source JAR:

  • javax/json
  • org/jdesktop

comment:55 Changed 9 months ago by simon04

In 16157/josm:

see #16860 - build.xml: ant sources: use javax.json:javax.json-api

comment:56 Changed 9 months ago by simon04

  • org/jdesktop is obsolete due to r16050

comment:57 Changed 9 months ago by simon04

In 16158/josm:

see #16860 - Apache Ivy: re-add org.glassfish:javax.json

comment:58 Changed 9 months ago by Don-vip

Summary: [PATCH] Remove svn:external for jmapviewer and use Apache Ivy to download jarRemove svn:external for jmapviewer and use Apache Ivy to download jar

comment:59 Changed 9 months ago by simon04

In 16165/josm:

see #16860 - Resolve PMD using Apache Ivy

comment:60 Changed 9 months ago by Don-vip

@Simon: if you begin to resolve tools via Ivy, please keep error_prone, no official version actually works, a lot of patches are needed. I'm also using a Jacoco snapshot (unpatched) for Java 15 compatibility, so please wait the next release for this one. All others can be retrieved from Ivy I think.

comment:61 Changed 9 months ago by Don-vip

Also: don't forget plugins. build-common.xml has references to these jar files.

comment:62 Changed 9 months ago by Don-vip

@sebastic: I guess you don't need any of these test/static analysis tools for Debian packaging?

comment:63 Changed 9 months ago by simon04

Summary: Remove svn:external for jmapviewer and use Apache Ivy to download jarResolve all dependencies and tools using Apache Ivy

comment:64 in reply to:  60 Changed 9 months ago by simon04

Replying to Don-vip:

@Simon: if you begin to resolve tools via Ivy, please keep error_prone, no official version actually works, a lot of patches are needed. I'm also using a Jacoco snapshot (unpatched) for Java 15 compatibility, so please wait the next release for this one. All others can be retrieved from Ivy I think.

Alright :)

comment:65 Changed 9 months ago by simon04

In 16166/josm:

see #16860 - Resolve SpotBugs using Apache Ivy

comment:66 Changed 9 months ago by simon04

In 16167/josm:

see #16860 - Separate tools/ivy.xml

comment:67 Changed 9 months ago by simon04

[o35377] - see #josm16860 - Resolve SpotBugs using Apache Ivy

comment:68 Changed 9 months ago by simon04

In 16168/josm:

see #16860 - Resolve JavaCC using Apache Ivy

comment:69 Changed 9 months ago by simon04

[o35378] - see #josm16860 - Resolve JavaCC using Apache Ivy

comment:70 Changed 9 months ago by simon04

In 16169/josm:

see #16860 - Resolve ProGuard using Apache Ivy

comment:71 Changed 9 months ago by simon04

In 16171/josm:

see #16860 - Resolve Checkstyle using Apache Ivy

comment:72 Changed 9 months ago by simon04

[o35379] - see #see16860 - Resolve Checkstyle using Apache Ivy

comment:73 in reply to:  62 Changed 9 months ago by sebastic

Replying to Don-vip:

@sebastic: I guess you don't need any of these test/static analysis tools for Debian packaging?

Only if they're called from build.xml via the dist target dependency chain.

comment:74 Changed 9 months ago by skyper

Not sure if this is the ticket to blame but there was no latest built this morning.

comment:75 in reply to:  74 Changed 9 months ago by Don-vip

Replying to skyper:

Not sure if this is the ticket to blame but there was no latest built this morning.

Yep, the server Makefile didn't like r16168. Looking into it

comment:76 Changed 9 months ago by Don-vip

Fixed. But r16165 broke ImageryCompare, looking into this now.

EDIT: fixed

Last edited 9 months ago by Don-vip (previous) (diff)

comment:77 Changed 9 months ago by Don-vip

In 16176/josm:

see #16860 - update Eclipse classpath

comment:78 Changed 9 months ago by Don-vip

@Simon: ant spotbugs is broken for plugins:

BUILD FAILED
/var/lib/jenkins/jobs/JOSM-Plugins/workspace/jdk/JDK8/build.xml:60: The following error occurred while executing this line:
/var/lib/jenkins/jobs/JOSM-Plugins/workspace/jdk/JDK8/build.xml:29: The following error occurred while executing this line:
/var/lib/jenkins/jobs/JOSM-Plugins/workspace/jdk/JDK8/build-common.xml:666: impossible to build ivy path: bad confs provided: spotbugs not found among [default]

comment:79 Changed 9 months ago by simon04

I cannot reproduce locally. So I'm groping in the dark. Maybe [o35389] helps.

[o35389] - see ​#josm16860 - Add ant resolve-tools target

[o35390] - see ​#josm16860 - Update to Apache Ivy 2.5.0

comment:80 Changed 9 months ago by simon04

I would like to get rid of source:trunk/tools/ivy/ivy.jar in the repository.

Variant 1: require Apache Ivy to be installed systemwide (as we do for Apache Ant)

Variant 2: download from https://josm.openstreetmap.de/nexus/content/repositories/public/ (as we do for plugins, as is shown in the most basic Ivy example)

  • build.xml

    diff --git a/build.xml b/build.xml
    index cc3a33e7a..11d2334c1 100644
    a b  
    1616         xmlns:unless="ant:unless"
    1717>
    1818    <target name="init-ivy">
     19        <property name="ivy.version" value="2.5.0"/>
    1920        <dirname property="base.dir" file="${ant.file.josm}"/>
    2021        <property name="lib.dir"   location="${base.dir}/lib"/>
    2122        <property name="tools.dir" location="${base.dir}/tools"/>
    2223        <property name="tools.ivy" location="${tools.dir}/ivy.xml"/>
    23         <taskdef resource="org/apache/ivy/ant/antlib.xml" uri="antlib:org.apache.ivy.ant" classpath="${tools.dir}/ivy/ivy.jar"/>
     24        <property name="ivy.jar.dir" location="${tools.dir}/ivy"/>
     25        <property name="ivy.jar.file" location="${ivy.jar.dir}/ivy-${ivy.version}.jar"/>
     26        <mkdir dir="${ivy.jar.dir}"/>
     27        <get src="https://josm.openstreetmap.de/nexus/content/repositories/public/org/apache/ivy/ivy/${ivy.version}/ivy-${ivy.version}.jar"
     28             dest="${ivy.jar.file}"
     29             skipexisting="true"
     30        />
     31        <echo>Apache Ivy ${ivy.jar.file}</echo>
     32        <taskdef resource="org/apache/ivy/ant/antlib.xml" uri="antlib:org.apache.ivy.ant" classpath="${ivy.jar.file}"/>
    2433    </target>
    2534    <target name="init-properties" depends="resolve">
    2635        <property environment="env"/>

Which variant should we take?

comment:81 Changed 9 months ago by Don-vip

I would say 2. Less disruptive for users. Probably as a new step that can be skipped for Debian build, or any system that provides ivy?
@sebastic your opinion?

comment:82 Changed 9 months ago by sebastic

No preference, the Debian package build won't call the ivy target.

comment:83 in reply to:  79 Changed 9 months ago by Don-vip

Replying to simon04:

I cannot reproduce locally. So I'm groping in the dark. Maybe [o35389] helps.

[o35389] - see ​#josm16860 - Add ant resolve-tools target

[o35390] - see ​#josm16860 - Update to Apache Ivy 2.5.0

It's... different :)

resolve-tools:

BUILD FAILED
/var/lib/jenkins/jobs/JOSM-Plugins/workspace/jdk/JDK8/build.xml:57: The following error occurred while executing this line:
/var/lib/jenkins/jobs/JOSM-Plugins/workspace/jdk/JDK8/build.xml:29: The following error occurred while executing this line:
/var/lib/jenkins/jobs/JOSM-Plugins/workspace/jdk/JDK8/build-common.xml:727: java.lang.NullPointerException
	at org.apache.ivy.ant.IvyAntSettings.getDefaultProperties(IvyAntSettings.java:322)
	at org.apache.ivy.ant.IvyAntSettings$1.execute(IvyAntSettings.java:268)
	at org.apache.ivy.ant.IvyAntSettings.createIvyEngine(IvyAntSettings.java:273)
	at org.apache.ivy.ant.IvyAntSettings.getDefaultInstance(IvyAntSettings.java:139)
	at org.apache.ivy.ant.IvyAntSettings.getDefaultInstance(IvyAntSettings.java:150)
	at org.apache.ivy.ant.IvyTask.getIvyInstance(IvyTask.java:77)
	at org.apache.ivy.ant.IvyTask.prepareTask(IvyTask.java:237)
	at org.apache.ivy.ant.IvyResolve.prepareTask(IvyResolve.java:236)
	at org.apache.ivy.ant.IvyTask.execute(IvyTask.java:258)
	at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:292)
	at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:99)
	at org.apache.tools.ant.Task.perform(Task.java:350)
	at org.apache.tools.ant.Target.execute(Target.java:449)

comment:84 Changed 9 months ago by Don-vip

The JDK11 build passed, I'm relaunching the JDK8 build.

comment:85 Changed 9 months ago by simon04

In 16183/josm:

see #16860 - <get> Apache Ivy, remove ivy.jar from repository

comment:86 in reply to:  49 Changed 9 months ago by simon04

Replying to simon04:

I've asked the apache-jcs maintainers to craft a new release which we could use via Apache Ivy: [JCS-204] Craft a new 2.3 release - https://issues.apache.org/jira/browse/JCS-204

Thomas Vandahl commented on JCS-204:

Unfortunately, the current trunk/master contains lots of breaking changes, so that a new release *must* be 3.x. This will require the package name as well as the Maven coordinates to reflect this. I'll see what I can do.

Update: I renamed the ticket to [JCS-204] Release JCS 3.0 - ASF JIRA

Last edited 9 months ago by simon04 (previous) (diff)

comment:87 Changed 9 months ago by simon04

@Vincent, how should we proceed with commons-jcs and jmapviewer?

Ad commons-jcs: could we publish the current SNAPSHOT to JOSM's Nexus and use 3.0-SNAPSHOT from there?

Ad jmapviewer: the latest release in JOSM's Nexus is 2.9, the latest released version 2.13 – Can we publish 2.10…2.13 to Nexus and include 2.13 via Apache Ivy

comment:88 in reply to:  87 ; Changed 9 months ago by Don-vip

Replying to simon04:

@Vincent, how should we proceed with commons-jcs and jmapviewer?

Ad commons-jcs: could we publish the current SNAPSHOT to JOSM's Nexus and use 3.0-SNAPSHOT from there?

Is 3.0-SNAPSHOT already available?

Ad jmapviewer: the latest release in JOSM's Nexus is 2.9, the latest released version 2.13 – Can we publish 2.10…2.13 to Nexus and include 2.13 via Apache Ivy

The jmapviewer nexus deploy currently relies on a very unreliable Jenkins job: https://josm.openstreetmap.de/jenkins/job/Nexus-JMapViewer/
We should make something most robust. Ideally from Ant? Something like an "ant release" target? For the old releases I can import them manually.

comment:89 in reply to:  88 Changed 9 months ago by Don-vip

Replying to Don-vip:

The jmapviewer nexus deploy currently relies on a very unreliable Jenkins job: https://josm.openstreetmap.de/jenkins/job/Nexus-JMapViewer/
We should make something most robust. Ideally from Ant? Something like an "ant release" target? For the old releases I can import them manually.

I have changed the Jenkins job to take VERSION as a parameter, seems to work. We can keep it is it until we switch to Gitlab I think. All versions should be available in Nexus now.

comment:90 in reply to:  88 ; Changed 9 months ago by simon04

Replying to Don-vip:

Is 3.0-SNAPSHOT already available?

3.0-SNAPSHOT is the version we are currently using via svn:externals, i.e., https://github.com/apache/commons-jcs/commit/366e4f001aaa72cf489f25d642ff88648a81a3fe
Can we compile and publish this version to our Nexus?

The jmapviewer nexus deploy currently relies on a very unreliable Jenkins job: https://josm.openstreetmap.de/jenkins/job/Nexus-JMapViewer/
We should make something most robust. Ideally from Ant? Something like an "ant release" target? For the old releases I can import them manually.

Thank you. Shall we migrate to jmapviewer 2.13 via Apache Ivy now?

Having accomplished those two steps would allow us to eliminate the svn:externals completely. :))

comment:91 in reply to:  90 Changed 9 months ago by Don-vip

Replying to simon04:

Can we compile and publish this version to our Nexus?

I setup an automatic build&deploy job: https://josm.openstreetmap.de/jenkins/job/Nexus-JCS/7/console
Can you please test?

Thank you. Shall we migrate to jmapviewer 2.13 via Apache Ivy now?

Sure!

Having accomplished those two steps would allow us to eliminate the svn:externals completely. :))

\o/

comment:92 Changed 9 months ago by simon04

In 16194/josm:

see #16860 - Resolve JMapViewer using Apache Ivy

comment:93 Changed 9 months ago by simon04

In 16195/josm:

see #16860 - Resolve commons-jcs-core using Apache Ivy

comment:94 Changed 9 months ago by simon04

Awesome, thank you! Seems to be working like a charm (at least on my system). In r16195 I had to remove the <exclude>s (i.e., not migrate them to extract-libraries) as otherwise spurious JCS/Cache exceptions would show up.

comment:95 Changed 9 months ago by Don-vip

Proguard fails.

comment:96 Changed 9 months ago by simon04

In 16197/josm:

see #16860 - Apache Ivy: exclude various commons-jcs classes from inclusion

comment:97 in reply to:  95 Changed 9 months ago by simon04

Replying to Don-vip:

Proguard fails.

The usual suspect ;)

comment:98 Changed 9 months ago by skyper

Is this ticket to blame that there was no latest released this morning ?

comment:99 in reply to:  98 Changed 8 months ago by anonymous

Replying to skyper:

Is this ticket to blame that there was no latest released this morning ?

no latest today, should be 16199.

comment:100 Changed 8 months ago by Don-vip

Ticket not to blame, random network error. Tomorrow it should work.

comment:101 Changed 8 months ago by simon04

In 16209/josm:

see #16860 - directory cleanup

comment:102 Changed 8 months ago by GerdP

After updating to r16211 I can no longer compile in Eclipse. I tried

ant clean dist

and a refresh in Eclipse but that did not help. I've also restarted Eclipse. I still have 1309 compile errors.

comment:103 Changed 8 months ago by GerdP

Seems all sources containing import org.openstreetmap.gui.* fail.

comment:104 Changed 8 months ago by GerdP

Fixed, had to do ivy resolve manually in Eclipse

comment:105 in reply to:  104 ; Changed 8 months ago by Bjoeni

Disclaimer: I didn't really follow / read all of this ticket and I never worked with Apache Ivy before (I only stumbled across this ticket because org.openstreetmap.gui.* failed to resolve).

But what's the best way to use JMapViewer in a JOSM plugin at the moment? I installed IvyDE in Eclipse and had it resolve, it added a reference to jmapviewer-2.13.jar. I then added that jar (from the Ivy cache directory) manually to the referenced libraries of my plugin. Is that the way to go as of right now or am I just not getting how Ivy is supposed to work?

comment:106 Changed 8 months ago by Klumbumbus

In 16219/josm:

see #16860, see #18880 - Adapt Netbeans config to r16165

comment:107 Changed 8 months ago by Klumbumbus

After r16166 Netbeans can't find spotbugs.jar anymore and the build fails, probably also due to r16194, as the message is:

C:\Users\stefa\Documents\OSM\josm\core\ide\netbeans\nbbuild.xml:90: C:\Users\stefa\Documents\OSM\josm\core\src\org\openstreetmap\gui\jmapviewer\images does not exist.
BUILD FAILED (total time: 1 minute 34 seconds)

(see also #18880)

comment:108 in reply to:  105 Changed 8 months ago by taylor.smock

Replying to Bjoeni:

Disclaimer: I didn't really follow / read all of this ticket and I never worked with Apache Ivy before (I only stumbled across this ticket because org.openstreetmap.gui.* failed to resolve).

But what's the best way to use JMapViewer in a JOSM plugin at the moment? I installed IvyDE in Eclipse and had it resolve, it added a reference to jmapviewer-2.13.jar. I then added that jar (from the Ivy cache directory) manually to the referenced libraries of my plugin. Is that the way to go as of right now or am I just not getting how Ivy is supposed to work?

I've had much the same issue. This (also) probably isn't the best way to go about it, but I've been changing the build path. So "Configure build path" -> "Order and Export" -> Check "Ivy". I've done this since most of my projects use JSON, and javax.json is now resolved by Ivy.

If you do that, you will have a diff for .classpath that should look like this:

  • .classpath

     
    3232                        <attribute name="test" value="true"/>
    3333                </attributes>
    3434        </classpathentry>
    35         <classpathentry kind="con" path="org.apache.ivyde.eclipse.cpcontainer.IVYDE_CONTAINER/?project=JOSM&amp;ivyXmlPath=ivy.xml&amp;confs=*&amp;ivySettingsPath=ivysettings.xml&amp;loadSettingsOnDemand=false&amp;ivyUserDir=&amp;propertyFiles="/>
     35        <classpathentry exported="true" kind="con" path="org.apache.ivyde.eclipse.cpcontainer.IVYDE_CONTAINER/?project=JOSM&amp;ivyXmlPath=ivy.xml&amp;confs=*&amp;ivySettingsPath=ivysettings.xml&amp;loadSettingsOnDemand=false&amp;ivyUserDir=&amp;propertyFiles="/>
    3636        <classpathentry kind="lib" path="test/lib/jfcunit.jar">
    3737                <attributes>
    3838                        <attribute name="test" value="true"/>

comment:109 Changed 8 months ago by Don-vip

In 16221/josm:

see #16860 - remove copy of jmapviewer images in Netbeans project

comment:110 Changed 8 months ago by Don-vip

@Klumbumbus can you please test Netbeans?

comment:111 Changed 8 months ago by Klumbumbus

The build works again, but it still can't find spotbugs.jar.

comment:112 Changed 8 months ago by Don-vip

In 16222/josm:

see #16860 - update spotbugs jar in Netbeans project

comment:113 Changed 8 months ago by Don-vip

Done. Please check.

comment:114 Changed 8 months ago by Klumbumbus

👍

comment:115 Changed 8 months ago by Don-vip

Resolution: fixed
Status: newclosed

comment:116 Changed 8 months ago by Don-vip

Resolution: fixed
Status: closedreopened

Something happened between r16155 and r16200 that broke the build for Java 14/15, but not for Java 8/11.
This is the output we get with Java 14+:

test-compile:
    [javac] Compiling 569 source files to /var/lib/jenkins/jobs/Java-EarlyAccess-JOSM/workspace/jdk/JDK14/test/build/unit
    [javac] warning: [options] bootstrap class path not set in conjunction with -source 8
    [javac] 1 warning
    [javac] Compiling 19 source files to /var/lib/jenkins/jobs/Java-EarlyAccess-JOSM/workspace/jdk/JDK14/test/build/functional
    [javac] warning: [options] bootstrap class path not set in conjunction with -source 8
    [javac] 1 warning
    [javac] Compiling 12 source files to /var/lib/jenkins/jobs/Java-EarlyAccess-JOSM/workspace/jdk/JDK14/test/build/performance
    [javac] warning: [options] bootstrap class path not set in conjunction with -source 8
    [javac] 1 warning

test:
     [echo] Running unit tests with JUnit
[jacoco:coverage] Enhancing junit with coverage
    [junit] Error: A JNI error has occurred, please check your installation and try again
    [junit] Running org.openstreetmap.josm.tools.template_engine.Batch-With-Multiple-Tests
    [junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0 sec
    [junit] Tests FAILED (crashed)
     [echo] Running functional tests with JUnit
[jacoco:coverage] Enhancing junit with coverage
    [junit] Error: A JNI error has occurred, please check your installation and try again
    [junit] Running org.openstreetmap.josm.tools.Batch-With-Multiple-Tests
    [junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0 sec
    [junit] Tests FAILED (crashed)

comment:117 Changed 8 months ago by simon04

I cannot reproduce locally with r16225

$ gradle --version
...
Ant:          Apache Ant(TM) version 1.10.7 compiled on September 1 2019
JVM:          14 (AdoptOpenJDK 14+36)

$ ant clean

$ ant test '-Ddefault-junit-includes=**/josm/tools/**/*Test.class'
...
test:
     [echo] Running unit tests with JUnit
...
BUILD SUCCESSFUL
Total time: 1 minute 11 seconds

comment:118 in reply to:  60 ; Changed 8 months ago by simon04

Replying to Don-vip:

@Simon: if you begin to resolve tools via Ivy, please keep error_prone, no official version actually works, a lot of patches are needed. I'm also using a Jacoco snapshot (unpatched) for Java 15 compatibility, so please wait the next release for this one.

Can we deploy those snapshots to https://josm.openstreetmap.de/nexus/content/repositories/snapshots/ as we've done for Apache Commons JCS? Then we could also resolve all test dependencies using Apache Ivy.

JaCoCo got official Java 14 support in the most recent commit (with the nice commit message "Happy birthday JDK 14!"): https://github.com/jacoco/jacoco/commit/1c5601bba5bf195a5f9b09a87cb5d2152857bbb7#diff-0886964cfc7b181768d1d9b182f50528R25

comment:119 Changed 8 months ago by Don-vip

Ah you're right it's just that I enabled Jacoco coverage in r16156. Strangely it works also for me locally, even in headless mode.

comment:120 in reply to:  118 ; Changed 8 months ago by Don-vip

Replying to simon04:

Can we deploy those snapshots to https://josm.openstreetmap.de/nexus/content/repositories/snapshots/ as we've done for Apache Commons JCS?

No need, they have their own maven repo for snapshots.

Then we could also resolve all test dependencies using Apache Ivy.

I'm trying to do it :)

comment:121 in reply to:  120 Changed 8 months ago by Don-vip

Replying to Don-vip:

I'm trying to do it :)

I have bad luck: the very moment I try to use their snapshots repository, it goes down: https://status.maven.org/incidents/bwklmrfj9z1c

comment:122 in reply to:  120 Changed 8 months ago by simon04

Replying to Don-vip:

No need, they have their own maven repo for snapshots.

Convenient :-)

Replying to Don-vip:

I have bad luck: the very moment I try to use their snapshots repository, it goes down

:-D
It's up again:

Quoting https://status.maven.org/incidents/bwklmrfj9z1c

Services have been restored on oss.sonatype.org.
Posted 11 minutes ago. Apr 04, 2020 - 09:20 EDT

comment:123 Changed 8 months ago by Don-vip

In 16229/josm:

see #16860 - resolve test libraries using Ivy

comment:124 Changed 8 months ago by Don-vip

In 16231/josm:

see #16860 - fix compilation of scripts

comment:125 Changed 8 months ago by Don-vip

Priority: normalmajor

comment:126 Changed 8 months ago by Don-vip

Resolution: fixed
Status: reopenedclosed

Found out what the Java 14 issue was. It came from the Jenkins job configuration:

jacoco.includes=org.openstreetmap.josm.*:java.*:javax.*:com.sun.*:sun.*
jacoco.inclbootstrapclasses=true
jacoco.inclnolocationclasses=true

I think I tried a long time ago to cover JDK classes too, when we were part of the OpenJDK Outreach, before Oracle decided to go full $$$ on Java.
I removed this configuration and now it works :)

comment:127 Changed 8 months ago by Don-vip

Plugin build updated in [o35411:35412].

Last edited 8 months ago by Don-vip (previous) (diff)

comment:128 Changed 8 months ago by simon04

[o35413] - resolve test libraries using Ivy settings from core

comment:129 Changed 8 months ago by GerdP

Is there an easy way to suppress all the [ivy:resolve] messages?
When I use e.g.

ant checkstyle pmd spotbugs

it is difficult to find the messages from checkstyle and checkstyle-josm.xml is hard to read because of its length.
I also wonder why I see those for each target. Is that intended?
For now I try to get used to a different order:

ant pmd spotbugs checkstyle 

comment:130 Changed 8 months ago by simon04

In 16243/josm:

see #16860 - Apache Ivy: tune logging

comment:131 Changed 8 months ago by GerdP

thanks!

comment:132 in reply to:  41 Changed 8 months ago by sebastic

Replying to simon04:

The josm-sources.jar would contain all JOSM's own sources (.java files + resources) and all of its dependencies (plenty of additional .java files + resources)? This seems doable, but requires some investigations into Ant/Ivy (at least, I don't know how to accomplish this by heart).

The current source JAR is not sufficient, there are some transitive dependencies, to build the subset of metadata-extractor required for JOSM the adobe-xmp-core sources are also required, but it's license terms are non-free i.e. incompatible with the DFSG & OSD.

The OpeningHoursParser sources also cannot be built because the ParseException & Token symbols cannot be found. This seems to be missing imports. It may also be related to using JDK 11 to build JOSM whereas OpeningHoursParser likely still uses JDK 8.

Update: Compiling the metadata-extractor sources was possible after adding the sources from xmpcore-6.0.6-sources.jar (BSD-3-Clause), and compiling the OpeningHoursParser sources after adding the sources from annotations-19.0.0-sources.jar (Apache-2.0) and a few generated sources.

Last edited 8 months ago by sebastic (previous) (diff)

comment:133 Changed 8 months ago by simon04

In 16338/josm:

see #16860 - Apache Ivy: update xmpcore to fix warning

comment:134 Changed 8 months ago by simon04

In 16339/josm:

see #16860 - Apache Ivy: remove unused epsg configuration

comment:135 Changed 8 months ago by GerdP

Please check https://josm.openstreetmap.de/jenkins/job/JOSM-Plugins/jdk=JDK8/1842/console

BUILD FAILED
/var/lib/jenkins/jobs/JOSM-Plugins/workspace/jdk/JDK8/build.xml:54: The following error occurred while executing this line:
/var/lib/jenkins/jobs/JOSM-Plugins/workspace/jdk/JDK8/build.xml:29: The following error occurred while executing this line:
/var/lib/jenkins/jobs/JOSM-Plugins/workspace/jdk/JDK8/build-common.xml:551: impossible to ivy retrieve: java.lang.RuntimeException: problem during retrieve of org.openstreetmap.josm.plugins#wikipedia: java.lang.IllegalStateException: Report file '/var/lib/jenkins/.ant/cache/org.openstreetmap.josm.plugins-wikipedia-test.xml' does not exist.

comment:136 Changed 7 months ago by stoecker

Build in plugins fails:

~/josm/plugins/apache-commons> ant
Buildfile: /home/stoecker/sources/svn.openstreetmap.org/applications/editors/josm/plugins/apache-commons/build.xml

resolve:

BUILD FAILED
/home/stoecker/sources/svn.openstreetmap.org/applications/editors/josm/plugins/apache-commons/build.xml:23: Problem: failed to create task or type antlib:org.apache.ivy.ant:retrieve
Cause: The name is undefined.
Action: Check the spelling.
Action: Check that any custom tasks/types have been declared.
Action: Check that any <presetdef>/<macrodef> declarations have taken place.
No types or tasks have been defined in this namespace yet

This appears to be an antlib declaration. 
Action: Check that the implementing library exists in one of:
        -/usr/share/ant/lib
        -/home/stoecker/.ant/lib
        -a directory added on the command line with the -lib argument


Total time: 0 seconds

comment:137 Changed 7 months ago by stoecker

Resolution: fixed
Status: closedreopened

comment:138 Changed 7 months ago by simon04

In 16403/josm:

see #16860 - Resolve oauth.signpost:signpost-core via Apache Ivy

Signpost 2.0.0 includes the update to Java 8: https://github.com/mttkay/signpost/releases/tag/oauth-signpost-2.0.0

comment:139 Changed 7 months ago by GerdP

plugin pdfimport is missing org.apache.commons.logging and others:
https://josm.openstreetmap.de/jenkins/job/JOSM-Plugins/jdk=JDK8/1850/consoleText
I assume something reg. ivy is missing?

comment:140 Changed 7 months ago by GerdP

I can compile the plugin when I uncomment the line in the build.xml

<!-- <property name="plugin.requires" value="apache-commons"/> -->

Why is this line needed again now? It was commented in 2018 and the build worked in April 2020

comment:141 Changed 7 months ago by taylor.smock

Possibly. It looks like there used to be a src/org/apache/commons/logging directory in the source code, which is why the plugin was compiling correctly (the logging library removed in r16050).

Do you know if there was a reason for using a logging library directly instead of JOSM Logging?

comment:142 Changed 7 months ago by GerdP

I don't understand what's going on. I've reverted my changes and now ant clean dist works in directory pdfimport.

comment:143 Changed 7 months ago by GerdP

.. and now its back again. Seems a dependency is missing?

comment:144 Changed 7 months ago by simon04

commons-logging via Ivy: r16024
commons-logging dropped as it is nowhere used in JOSM core: r16397

comment:145 Changed 7 months ago by GerdP

So, what should be done with plugins? Is

<property name="plugin.requires" value="apache-commons"/> 

the solution?

comment:146 Changed 7 months ago by taylor.smock

It is probably the quickest solution.

But all the pdfimport plugin is doing is creating a log instance (with Apache Logging LogFactory), and then logging exceptions to it (log.warn). I think that pdfimport can, at least, get by without using the Apache Logging library. I think that the loggers could be replaced with Logging.warn(e) with no loss of functionality.

comment:147 Changed 7 months ago by simon04

pdfimport: removed commong.logging usages in [o35448:35449].

However, pdfbox internally uses log4j according to https://mvnrepository.com/artifact/org.apache.pdfbox/pdfbox/1.8.12 and the following warning is printed (I can live with that):

log4j:WARN No appenders could be found for logger (org.apache.pdfbox.pdfparser.PDFObjectStreamParser).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

comment:148 Changed 7 months ago by simon04

In 16408/josm:

see #16860 - build.xml: remove leftovers of org.apache.commons.logging

comment:149 Changed 6 months ago by Don-vip

In 16536/josm:

see #16860 - remove leftovers of OAuth signpost sources

comment:150 Changed 6 months ago by Don-vip

Can't we remove com/google/gdata/util/common/base? According to r4231 it seems used by signpost and/or metadata extractor, both are managed by Ivy now.

comment:151 Changed 6 months ago by Don-vip

Milestone: 20.0320.05

comment:152 Changed 6 months ago by simon04

In 16537/josm:

see #16860 - Remove src/com/google/gdata

Leftover from r16403

comment:153 Changed 6 months ago by Don-vip

Only one dependency to go \o/

comment:154 Changed 6 months ago by simon04

The hardest one … Might not be ready for the 20.05 milestone … ;-)

Should we release r16538 as 20.05?

comment:155 in reply to:  154 ; Changed 6 months ago by Don-vip

Milestone: 20.0520.06

Replying to simon04:

The hardest one … Might not be ready for the 20.05 milestone … ;-)

Should we release r16538 as 20.05?

If you feel it's stable enough, let's do it. I didn't follow JOSM development on last month.

comment:156 in reply to:  155 Changed 6 months ago by stoecker

Replying to Don-vip:

Replying to simon04:

The hardest one … Might not be ready for the 20.05 milestone … ;-)

Should we release r16538 as 20.05?

If you feel it's stable enough, let's do it. I didn't follow JOSM development on last month.

You have to wait till tomorrow, as this version lies behind the nightly build. I've not seen anything major in the new tickets lately, so I also have no objections. I actually wanted to set sundays version but there were new checkins all the time.

comment:157 Changed 6 months ago by simon04

  • r16536 is the built latest version.
  • r16537 is irrelevant for the release.
  • r16538 is an i18n update of a few languages.

If we can trigger a latest build for r16538, let's take this one. Otherwise r16536 is good. :-)

Thanks!

comment:158 Changed 6 months ago by stoecker

We waited so long, we can wait another day :-)

comment:159 Changed 6 months ago by Don-vip

I just launched a manual build :)

comment:160 Changed 6 months ago by simon04

In 16603/josm:

see #16860 - Suppress "bad path element .../xmpcore-6.0.6.jar: no such file or directory" warning

comment:161 Changed 6 months ago by simon04

Milestone: 20.06Longterm

comment:162 Changed 8 weeks ago by simon04

In 17128/josm:

see #16860 - Apache Ivy: only use josm-nexus repository by default

comment:163 Changed 5 weeks ago by GerdP

What are the required steps to configure ivy so that it doesn't print lots of these error messages when compiling a plugin like e.g.. reverter?
[ivy:resolve] unknown resolver josm-nexus

comment:164 Changed 5 weeks ago by GerdP

In the wikipedia plugin directory I find this ivy_settings.xml :

<?xml version="1.0" encoding="utf-8"?>
<ivysettings>
  <settings defaultResolver="josm-nexus"/>
  <resolvers>
    <ibiblio name="josm-nexus" m2compatible="true" root="https://josm.openstreetmap.de/nexus/content/repositories/public/" />
  </resolvers>
</ivysettings>

When I copy this file into the reverter directory the messages disappear. Does that mean we need this file in every plugin directory that doesn't yet have one? Or is it possible to have a working default for all of them?

Modify Ticket

Change Properties
Set your email in Preferences
Action
as reopened The owner will remain team.
as The resolution will be set.
to The owner will be changed from team to the specified user.
The owner will change to wiktorn
as duplicate The resolution will be set to duplicate.The specified ticket will be cross-referenced with this ticket

Add Comment


E-mail address and name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.