Modify

Opened 18 months ago

Closed 18 hours ago

#16860 closed enhancement (fixed)

Resolve all dependencies and tools using Apache Ivy

Reported by: wiktorn Owned by: team
Priority: normal Milestone: 20.03
Component: Core Version:
Keywords: hack-weekend-2018-10 Cc: stoecker, sebastic

Description (last modified by wiktorn)

Follow up to: #8269 and #16420. Needed for: #16858

Use JMapViewer from our Nexus instead of use of svn:externals. This allows us to:

  • test in the field, how well does it's job in controlling our dependencies
  • move JMapViewer outside OSM SVN server

The goal is to do incremental changes and provide better dependency management than following HEAD on svn:externals.

Sponsored by: OpenStreetMap Hack Weekend, Karlsruhe Oct'18

Attachments (2)

ivy-jmapviewer.patch (13.8 KB) - added by wiktorn 18 months ago.
ivy-jmapviewer-v2.patch (12.3 KB) - added by simon04 15 months ago.

Download all attachments as: .zip

Change History (117)

Changed 18 months ago by wiktorn

Attachment: ivy-jmapviewer.patch added

comment:1 Changed 18 months ago by wiktorn

Description: modified (diff)
Keywords: hack-weekend-2018-10 added

comment:2 Changed 17 months ago by Don-vip

Milestone: 18.1018.11

comment:3 Changed 16 months ago by Don-vip

Milestone: 18.1118.12

comment:4 Changed 15 months ago by Don-vip

Milestone: 18.1219.01

Changed 15 months ago by simon04

Attachment: ivy-jmapviewer-v2.patch added

comment:5 Changed 15 months ago by simon04

I merged the patch with the recent changes and uploaded attachment:ivy-jmapviewer-v2.patch

Is there anything holding us back?

comment:6 in reply to:  5 Changed 15 months ago by stoecker

Is there anything holding us back?

Yes. The new server. We don't want to install and do anything new on the old one.

comment:7 Changed 14 months ago by Don-vip

Milestone: 19.0119.02

comment:8 Changed 13 months ago by Don-vip

Milestone: 19.0219.03

comment:9 Changed 12 months ago by Don-vip

Milestone: 19.0319.04

comment:10 Changed 11 months ago by Don-vip

Milestone: 19.0419.05

comment:11 Changed 11 months ago by Don-vip

Milestone: 19.05

comment:12 Changed 2 months ago by simon04

Milestone: 20.02

Since we got a new server in the meanwhile, is anything (else) holding us back?

comment:13 in reply to:  12 Changed 2 months ago by Don-vip

Replying to simon04:

is anything (else) holding us back?

I don't know yet how to complete the GitLab setup. I'll comment why in #16857 comments.

comment:14 Changed 6 weeks ago by simon04

Milestone: 20.0220.03

comment:15 Changed 4 weeks ago by simon04

In 15977/josm:

see #16860, see #18140 - Setup Apache Ivy (patch by wiktorn, modified)

comment:16 Changed 4 weeks ago by simon04

In 16011/josm:

see #16860, see #18140 - Restrict SpotBugs to org/openstreetmap/josm

comment:17 Changed 4 weeks ago by Don-vip

In 16020/josm:

see #16860, see #18140 - better way to restrict SpotBugs analysis scope (without error about missing classes)

See https://github.com/spotbugs/spotbugs/blob/4.0.0/spotbugs-ant/src/main/java/edu/umd/cs/findbugs/anttask/FindBugsTask.java

comment:18 Changed 4 weeks ago by simon04

In 16021/josm:

see #16860 - Resolve org.glassfish:javax.json via Apache Ivy

comment:19 Changed 4 weeks ago by simon04

In 16022/josm:

see #16860 - Resolve org.tukaani:xz via Apache Ivy

comment:20 Changed 4 weeks ago by simon04

In 16023/josm:

see #16860 - Resolve org.apache.commons:commons-compress via Apache Ivy

comment:21 Changed 4 weeks ago by simon04

In 16024/josm:

see #16860 - Resolve commons-logging:commons-logging via Apache Ivy

comment:22 Changed 4 weeks ago by simon04

In 16025/josm:

see #16860 - Resolve com.drewnoakes:metadata-extractor via Apache Ivy

comment:23 in reply to:  22 ; Changed 4 weeks ago by Don-vip

Replying to simon04:

In 16025/josm:

see #16860 - Resolve com.drewnoakes:metadata-extractor via Apache Ivy

Are you sure it works for this one? I remember I must patch it at each update.

comment:24 Changed 4 weeks ago by simon04

Here's how the dist sizes evolved:

ls -l --time-style full-iso 
total 84764
-rw-r--r-- 1 simon users 14443614 2020-03-03 22:21:13 +0100 josm-custom-16016.jar
-rw-r--r-- 1 simon users 14444368 2020-03-03 22:23:10 +0100 josm-custom-16016+json.jar
-rw-r--r-- 1 simon users 14429230 2020-03-03 22:24:45 +0100 josm-custom-16016+json+xz.jar
-rw-r--r-- 1 simon users 14432267 2020-03-03 22:37:35 +0100 josm-custom-16016+json+xz+compress.jar
-rw-r--r-- 1 simon users 14430688 2020-03-03 22:40:41 +0100 josm-custom-16016+json+xz+compress+logging.jar
-rw-r--r-- 1 simon users 14604031 2020-03-03 23:28:36 +0100 josm-custom-16016+json+xz+compress+logging+metadata-extractor.jar

The increase for metadata-extractor is due to our disabled imports in JpegMetadataReader.java which cannot be ported to Apache Ivy.


How to proceed?

comment:25 in reply to:  23 ; Changed 4 weeks ago by simon04

Replying to Don-vip:

Are you sure it works for this one? I remember I must patch it at each update.

Haven't found any problem yet. I successfully opened a bunch of JPG images; ExifReaderTest passes.

comment:26 in reply to:  24 Changed 4 weeks ago by Don-vip

Replying to simon04:

The increase for metadata-extractor is due to our disabled imports in JpegMetadataReader.java which cannot be ported to Apache Ivy.

We could ask the author to provide smaller artifacts.

comment:27 in reply to:  24 ; Changed 4 weeks ago by Don-vip

Replying to simon04:

  • signpost: we did some minor cleanups and kicked out commons-codecs in r8149, r10746

Signpost is dead, no commit in 5 years, 15 open pull requests...

Did we consider https://github.com/scribejava/scribejava at some point? It looks way more active, but probably heavier.

comment:28 in reply to:  25 ; Changed 4 weeks ago by Don-vip

Replying to simon04:

Replying to Don-vip:

Are you sure it works for this one? I remember I must patch it at each update.

Haven't found any problem yet. I successfully opened a bunch of JPG images; ExifReaderTest passes.

I'm almost sure it doesn't work. Look at proguard output:

 [proguard] Warning: com.drew.imaging.FileTypeDetector: can't find referenced class com.drew.imaging.quicktime.QuickTimeTypeChecker
 [proguard] Warning: com.drew.imaging.FileTypeDetector: can't find referenced class com.drew.imaging.quicktime.QuickTimeTypeChecker
 [proguard] Warning: com.drew.imaging.FileTypeDetector: can't find referenced class com.drew.imaging.riff.RiffTypeChecker
 [proguard] Warning: com.drew.imaging.FileTypeDetector: can't find referenced class com.drew.imaging.riff.RiffTypeChecker
 [proguard] Warning: com.drew.imaging.FileTypeDetector: can't find referenced class com.drew.imaging.mp3.MpegAudioTypeChecker
 [proguard] Warning: com.drew.imaging.FileTypeDetector: can't find referenced class com.drew.imaging.mp3.MpegAudioTypeChecker
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.psd.PsdMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.png.PngMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.bmp.BmpMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.gif.GifMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.ico.IcoMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.pcx.PcxMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.webp.WebpMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.raf.RafMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.avi.AviMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.wav.WavMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.quicktime.QuickTimeMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.mp4.Mp4MetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.mp3.Mp3MetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.eps.EpsMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.heif.HeifMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.psd.PsdMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.png.PngMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.bmp.BmpMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.gif.GifMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.ico.IcoMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.pcx.PcxMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.webp.WebpMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.raf.RafMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.avi.AviMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.wav.WavMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.quicktime.QuickTimeMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.mp4.Mp4MetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.mp3.Mp3MetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.eps.EpsMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.heif.HeifMetadataReader
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$1: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamFactory
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$1: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamFactory
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$1: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamFactory
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$1: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamFactory
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$1: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamProvider
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$1: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamProvider
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$1: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamFactory
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$1: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamProvider
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$1: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamProvider
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$1: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamProvider
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$1: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamProvider
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$1: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamFactory
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$2: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamFactory
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$2: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamFactory
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$2: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamFactory
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$2: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamFactory
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$2: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamProvider
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$2: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamProvider
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$2: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamFactory
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$2: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamProvider
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$2: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamProvider
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$2: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamProvider
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$2: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamProvider
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$2: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamFactory
 [proguard] Warning: org.apache.commons.compress.compressors.xz.XZUtils$CachedAvailability: can't find referenced class org.apache.commons.compress.compressors.xz.XZUtils
 [proguard] Warning: there were 61 unresolved references to classes or interfaces.

comment:29 in reply to:  27 Changed 4 weeks ago by simon04

Replying to Don-vip:

Did we consider https://github.com/scribejava/scribejava at some point? It looks way more active, but probably heavier.

Half an hour ago, I've also found this library. Looks interesting. Will investigate in the next days. The first thing I noticed is that com.github.scribejava.core.extractors.AbstractOAuth1JSONTokenExtractor makes use of Jackson for JSON decoding, but we could probably provide our own implementation.

comment:30 in reply to:  28 Changed 4 weeks ago by simon04

Replying to Don-vip:

I'm almost sure it doesn't work. Look at proguard output:

This is the list of classes which are now included in josm.jar, but haven't been before:

$ comm -3 all-classes-in-josm.jar-before all-classes-in-josm.jar-after
        com/adobe/
        com/adobe/internal/
        com/adobe/internal/xmp/
        com/adobe/internal/xmp/impl/
        com/adobe/internal/xmp/impl/Base64.class
        com/adobe/internal/xmp/impl/ByteBuffer.class
        com/adobe/internal/xmp/impl/CountOutputStream.class
        com/adobe/internal/xmp/impl/FixASCIIControlsReader.class
        com/adobe/internal/xmp/impl/ISO8601Converter.class
        com/adobe/internal/xmp/impl/Latin1Converter.class
        com/adobe/internal/xmp/impl/ParameterAsserts.class
        com/adobe/internal/xmp/impl/ParseRDF.class
        com/adobe/internal/xmp/impl/ParseState.class
        com/adobe/internal/xmp/impl/QName.class
        com/adobe/internal/xmp/impl/Utils.class
        com/adobe/internal/xmp/impl/XMPDateTimeImpl.class
        com/adobe/internal/xmp/impl/XMPIteratorImpl$NodeIterator$1.class
        com/adobe/internal/xmp/impl/XMPIteratorImpl$NodeIteratorChildren.class
        com/adobe/internal/xmp/impl/XMPIteratorImpl$NodeIterator.class
        com/adobe/internal/xmp/impl/XMPIteratorImpl.class
        com/adobe/internal/xmp/impl/XMPMetaImpl$1.class
        com/adobe/internal/xmp/impl/XMPMetaImpl$2.class
        com/adobe/internal/xmp/impl/XMPMetaImpl.class
        com/adobe/internal/xmp/impl/XMPMetaParser.class
        com/adobe/internal/xmp/impl/XMPNode$1.class
        com/adobe/internal/xmp/impl/XMPNode.class
        com/adobe/internal/xmp/impl/XMPNodeUtils.class
        com/adobe/internal/xmp/impl/XMPNormalizer.class
        com/adobe/internal/xmp/impl/XMPSchemaRegistryImpl$1.class
        com/adobe/internal/xmp/impl/XMPSchemaRegistryImpl.class
        com/adobe/internal/xmp/impl/XMPSerializerHelper.class
        com/adobe/internal/xmp/impl/XMPSerializerRDF.class
        com/adobe/internal/xmp/impl/XMPUtilsImpl.class
        com/adobe/internal/xmp/impl/xpath/
        com/adobe/internal/xmp/impl/xpath/PathPosition.class
        com/adobe/internal/xmp/impl/xpath/XMPPath.class
        com/adobe/internal/xmp/impl/xpath/XMPPathParser.class
        com/adobe/internal/xmp/impl/xpath/XMPPathSegment.class
        com/adobe/internal/xmp/options/
        com/adobe/internal/xmp/options/AliasOptions.class
        com/adobe/internal/xmp/options/IteratorOptions.class
        com/adobe/internal/xmp/options/Options.class
        com/adobe/internal/xmp/options/ParseOptions.class
        com/adobe/internal/xmp/options/PropertyOptions.class
        com/adobe/internal/xmp/options/SerializeOptions.class
        com/adobe/internal/xmp/properties/
        com/adobe/internal/xmp/properties/XMPAliasInfo.class
        com/adobe/internal/xmp/properties/XMPProperty.class
        com/adobe/internal/xmp/properties/XMPPropertyInfo.class
        com/adobe/internal/xmp/utils/
        com/adobe/internal/xmp/utils/XMLStreamWriterFactory.class
        com/adobe/internal/xmp/utils/XMLStreamWriterImpl.class
        com/adobe/internal/xmp/XMPConst.class
        com/adobe/internal/xmp/XMPDateTime.class
        com/adobe/internal/xmp/XMPDateTimeFactory.class
        com/adobe/internal/xmp/XMPError.class
        com/adobe/internal/xmp/XMPException.class
        com/adobe/internal/xmp/XMPIterator.class
        com/adobe/internal/xmp/XMPMeta.class
        com/adobe/internal/xmp/XMPMetaFactory$1.class
        com/adobe/internal/xmp/XMPMetaFactory.class
        com/adobe/internal/xmp/XMPPathFactory.class
        com/adobe/internal/xmp/XMPSchemaRegistry.class
        com/adobe/internal/xmp/XMPUtils.class
        com/adobe/internal/xmp/XMPVersionInfo.class
        com/drew/imaging/FileType.class
        com/drew/imaging/FileTypeDetector.class
        com/drew/imaging/ImageMetadataReader$1.class
        com/drew/imaging/ImageMetadataReader.class
        com/drew/imaging/tiff/TiffMetadataReader.class
        com/drew/imaging/TypeChecker.class
        com/drew/lang/ByteConvert.class
        com/drew/lang/ByteTrie$ByteTrieNode.class
        com/drew/lang/ByteTrie.class
        com/drew/lang/ByteUtil.class
        com/drew/lang/Iterables.class
        com/drew/lang/KeyValuePair.class
        com/drew/lang/NullOutputStream.class
        com/drew/lang/RandomAccessFileReader.class
        com/drew/lang/RandomAccessStreamReader.class
        com/drew/lang/StreamUtil.class
        com/drew/metadata/adobe/
        com/drew/metadata/adobe/AdobeJpegDescriptor.class
        com/drew/metadata/adobe/AdobeJpegDirectory.class
        com/drew/metadata/adobe/AdobeJpegReader.class
        com/drew/metadata/file/FileTypeDescriptor.class
        com/drew/metadata/file/FileTypeDirectory.class
        com/drew/metadata/jfif/
        com/drew/metadata/jfif/JfifDescriptor.class
        com/drew/metadata/jfif/JfifDirectory.class
        com/drew/metadata/jfif/JfifReader.class
        com/drew/metadata/jfxx/
        com/drew/metadata/jfxx/JfxxDescriptor.class
        com/drew/metadata/jfxx/JfxxDirectory.class
        com/drew/metadata/jfxx/JfxxReader.class
        com/drew/metadata/Schema.class
        com/drew/metadata/xmp/
        com/drew/metadata/xmp/XmpDescriptor.class
        com/drew/metadata/xmp/XmpDirectory.class
        com/drew/metadata/xmp/XmpReader.class
        com/drew/metadata/xmp/XmpWriter.class

In particular, FileTypeDetector and ImageMetadataReader are present in this list. We could exclude all those …

Last edited 4 weeks ago by simon04 (previous) (diff)

comment:31 Changed 4 weeks ago by simon04

In 16026/josm:

see #16860 - Apache Ivy: exclude various subclasses from inclusion

comment:32 Changed 4 weeks ago by simon04

In 16027/josm:

see #16860 - Apache Ivy: exclude com/drew/imaging/{FileTypeDetector,ImageMetadataReader} from inclusion

comment:33 Changed 4 weeks ago by Don-vip

In 16050/josm:

see #18845 - see #16860 - directory cleanup

comment:34 in reply to:  27 Changed 4 weeks ago by simon04

Replying to Don-vip:

Replying to simon04:

  • signpost: we did some minor cleanups and kicked out commons-codecs in r8149, r10746

Signpost is dead, no commit in 5 years, 15 open pull requests...

I posted to the signpost mailing list: https://groups.google.com/forum/#!msg/signpost-users/QDccoNNTvpc/5pRpdLZyAAAJ

comment:35 Changed 3 weeks ago by simon04

Cc: sebastic added

Hi sebastic, this might affect the packaging for Debian. A few libraries are obtained as JAR via Apache Ivy, see source:trunk/ivy.xml

comment:36 Changed 3 weeks ago by sebastic

move JMapViewer outside OSM SVN server

Where is it going? There is no JMapViewer repo on GitHub yet.

The Debian package uses the ZIP releases from https://svn.openstreetmap.org/applications/viewer/jmapviewer/releases/.

In the ivy config I see metadata-extractor, we used to build the josm package with the separately packaged metadata-extractor but because its API is highly unstable updating always broke either josm or one of the other dependent packages.

If the sources for unstable dependencies like this won't be bundled with josm any more, we won't be able to keep to the josm package in Debian.

comment:37 Changed 3 weeks ago by simon04

The goal is to use the official JAR distributions for external dependencie, and not (partially) compile them within the JOSM build system.

The current status:

  • Apache Commons Compress ✔
  • Apache Commons JCS: TODO, needs new release, see comment:24
  • Apache Commons Logging ✔
  • SvgSalamander: UNLIKELY
  • Metadata Extractor ✔
  • Signpost: TODO, WIP, see comment:34
  • Jsonp ✔
  • OpeningHoursValidator ✔
  • JMapViewer: TODO (needs separate repository first)

Running ant dist depends on ant resolve (which runs Apache Ivy to fetch the dependencies) and bundles the dependencies in a fatjar. So, by default all dependencies are bundled into dist/josm-custom.jar.

comment:38 Changed 3 weeks ago by sebastic

ant resolve doesn't work in a Debian build environment as it doesn't have network, the dependencies need to have their source included in the source tree or an appropriate version needs to be packaged and installed in the build environment.

Experience has shown that using separately packaged metadata-extractor and svgsalamander made backporting JOSM tested for Debian stable a PITA. Once the embedded copies were used this was much less of a problem. The separately packaged JMapViewer has also broken freeplane on more than one occasion, forcing stable users to choose whether to have the josm backport installed (which requires a newer jmapviewer) or a working freeplane.

Because these recent changes make building JOSM from source in a Debian build environment unfeasible when backports are required as well, we'll have to remove the josm package from Debian and replace it with an installer that downloads the pre-built JAR. This is not very good either (because the JARs don't have hashes nor cryptographic signatures to verify the download with), but at least it will keep JOSM tested easily available to users of Debian stable (assuming the josm-installer package passes FTP master review).

comment:39 Changed 3 weeks ago by simon04

Is there a Debian compatible way not involving svn:externals? By no means, the goal was to make Debian packging harder/impossible. However, slowly getting rid of svn:externals allows us to move forward in the long-term goal #16871.

comment:40 Changed 3 weeks ago by sebastic

Publish a source jar that bundles the required versions of the dependencies?

comment:41 Changed 3 weeks ago by simon04

The josm-sources.jar would contain all JOSM's own sources (.java files + resources) and all of its dependencies (plenty of additional .java files + resources)? This seems doable, but requires some investigations into Ant/Ivy (at least, I don't know how to accomplish this by heart).

comment:42 in reply to:  38 ; Changed 3 weeks ago by wiktorn

Replying to sebastic:

Because these recent changes make building JOSM from source in a Debian build environment unfeasible when backports are required as well, we'll have to remove the josm package from Debian and replace it with an installer that downloads the pre-built JAR. This is not very good either (because the JARs don't have hashes nor cryptographic signatures to verify the download with), but at least it will keep JOSM tested easily available to users of Debian stable (assuming the josm-installer package passes FTP master review).

Jars published on JOSM page are signed and it is possibile to verify the signature. See: https://docs.oracle.com/en/java/javase/11/tools/jarsigner.html

Replying to simon04:

The josm-sources.jar would contain all JOSM's own sources (.java files + resources) and all of its dependencies (plenty of additional .java files + resources)? This seems doable, but requires some investigations into Ant/Ivy (at least, I don't know how to accomplish this by heart).

If I remember correctly, it was possibile to download sources in Ivy

comment:43 Changed 3 weeks ago by Don-vip

Another Ivy issue we need to fix: proxy support, see https://lists.openstreetmap.org/pipermail/josm-dev/2020-March/008288.html

comment:44 in reply to:  42 Changed 3 weeks ago by sebastic

Replying to wiktorn:

Jars published on JOSM page are signed and it is possibile to verify the signature. See: https://docs.oracle.com/en/java/javase/11/tools/jarsigner.html

That's not a equivalent to detached GPG signatures.

jarsigner is part of the JDK, installing the entire JDK to run JOSM is highly undesirable.

comment:45 Changed 3 weeks ago by Don-vip

In 16118/josm:

see #16860 - instruct Eclipse IvyDE plugin to use ivysettings.xml

comment:46 Changed 3 weeks ago by Don-vip

In 16125/josm:

see #16860 - see #18880 - explicit use of Ivy configuration files for Netbeans compatibility

comment:47 Changed 2 weeks ago by Don-vip

In 16139/josm:

see #16860 - see #18880 - setup Ivy in Netbeans project

comment:48 Changed 2 weeks ago by simon04

In 16141/josm:

see #16860 - build.xml: add ant sources

Generates jar file of JOSM source files and its dependencies.

comment:49 Changed 2 weeks ago by simon04

I've asked the apache-jcs maintainers to craft a new release which we could use via Apache Ivy: [JCS-204] Craft a new 2.3 release - https://issues.apache.org/jira/browse/JCS-204

comment:50 in reply to:  48 ; Changed 2 weeks ago by Don-vip

Replying to simon04:

In 16141/josm:

see #16860 - build.xml: add ant sources

Generates jar file of JOSM source files and its dependencies.

@sebastic: I guess you don't need the REVISION file?

@simon: Dropping the "create-revision" dependency would make it easier for me to include this new task in the server Makefile

comment:51 in reply to:  50 Changed 2 weeks ago by sebastic

Replying to Don-vip:

@sebastic: I guess you don't need the REVISION file?

The package already creates it itself:

https://salsa.debian.org/debian-gis-team/josm/-/blob/debian/0.0.svn15937+dfsg-1/debian/rules#L80

And it's patched to include Build-Name & Debian-Release:

https://salsa.debian.org/debian-gis-team/josm/-/blob/debian/0.0.svn15937+dfsg-1/debian/patches/00-build.patch#L39

comment:52 Changed 2 weeks ago by simon04

In 16150/josm:

see #16860 - build.xml: ant sources: remove REVISION

comment:53 Changed 2 weeks ago by Don-vip

@sebastic can you please check if https://josm.openstreetmap.de/download/josm-latest-sources.jar is enough for you to build a JOSM Debian package.

comment:54 Changed 2 weeks ago by sebastic

JOSM tested 15937 contained these subdirectories under src/ which are not included in the source JAR:

  • javax/json
  • org/jdesktop

comment:55 Changed 2 weeks ago by simon04

In 16157/josm:

see #16860 - build.xml: ant sources: use javax.json:javax.json-api

comment:56 Changed 2 weeks ago by simon04

  • org/jdesktop is obsolete due to r16050

comment:57 Changed 2 weeks ago by simon04

In 16158/josm:

see #16860 - Apache Ivy: re-add org.glassfish:javax.json

comment:58 Changed 2 weeks ago by Don-vip

Summary: [PATCH] Remove svn:external for jmapviewer and use Apache Ivy to download jarRemove svn:external for jmapviewer and use Apache Ivy to download jar

comment:59 Changed 2 weeks ago by simon04

In 16165/josm:

see #16860 - Resolve PMD using Apache Ivy

comment:60 Changed 2 weeks ago by Don-vip

@Simon: if you begin to resolve tools via Ivy, please keep error_prone, no official version actually works, a lot of patches are needed. I'm also using a Jacoco snapshot (unpatched) for Java 15 compatibility, so please wait the next release for this one. All others can be retrieved from Ivy I think.

comment:61 Changed 2 weeks ago by Don-vip

Also: don't forget plugins. build-common.xml has references to these jar files.

comment:62 Changed 2 weeks ago by Don-vip

@sebastic: I guess you don't need any of these test/static analysis tools for Debian packaging?

comment:63 Changed 2 weeks ago by simon04

Summary: Remove svn:external for jmapviewer and use Apache Ivy to download jarResolve all dependencies and tools using Apache Ivy

comment:64 in reply to:  60 Changed 2 weeks ago by simon04

Replying to Don-vip:

@Simon: if you begin to resolve tools via Ivy, please keep error_prone, no official version actually works, a lot of patches are needed. I'm also using a Jacoco snapshot (unpatched) for Java 15 compatibility, so please wait the next release for this one. All others can be retrieved from Ivy I think.

Alright :)

comment:65 Changed 2 weeks ago by simon04

In 16166/josm:

see #16860 - Resolve SpotBugs using Apache Ivy

comment:66 Changed 2 weeks ago by simon04

In 16167/josm:

see #16860 - Separate tools/ivy.xml

comment:67 Changed 2 weeks ago by simon04

[o35377] - see #josm16860 - Resolve SpotBugs using Apache Ivy

comment:68 Changed 2 weeks ago by simon04

In 16168/josm:

see #16860 - Resolve JavaCC using Apache Ivy

comment:69 Changed 2 weeks ago by simon04

[o35378] - see #josm16860 - Resolve JavaCC using Apache Ivy

comment:70 Changed 2 weeks ago by simon04

In 16169/josm:

see #16860 - Resolve ProGuard using Apache Ivy

comment:71 Changed 2 weeks ago by simon04

In 16171/josm:

see #16860 - Resolve Checkstyle using Apache Ivy

comment:72 Changed 2 weeks ago by simon04

[o35379] - see #see16860 - Resolve Checkstyle using Apache Ivy

comment:73 in reply to:  62 Changed 2 weeks ago by sebastic

Replying to Don-vip:

@sebastic: I guess you don't need any of these test/static analysis tools for Debian packaging?

Only if they're called from build.xml via the dist target dependency chain.

comment:74 Changed 2 weeks ago by skyper

Not sure if this is the ticket to blame but there was no latest built this morning.

comment:75 in reply to:  74 Changed 2 weeks ago by Don-vip

Replying to skyper:

Not sure if this is the ticket to blame but there was no latest built this morning.

Yep, the server Makefile didn't like r16168. Looking into it

comment:76 Changed 2 weeks ago by Don-vip

Fixed. But r16165 broke ImageryCompare, looking into this now.

EDIT: fixed

Last edited 2 weeks ago by Don-vip (previous) (diff)

comment:77 Changed 13 days ago by Don-vip

In 16176/josm:

see #16860 - update Eclipse classpath

comment:78 Changed 12 days ago by Don-vip

@Simon: ant spotbugs is broken for plugins:

BUILD FAILED
/var/lib/jenkins/jobs/JOSM-Plugins/workspace/jdk/JDK8/build.xml:60: The following error occurred while executing this line:
/var/lib/jenkins/jobs/JOSM-Plugins/workspace/jdk/JDK8/build.xml:29: The following error occurred while executing this line:
/var/lib/jenkins/jobs/JOSM-Plugins/workspace/jdk/JDK8/build-common.xml:666: impossible to build ivy path: bad confs provided: spotbugs not found among [default]

comment:79 Changed 11 days ago by simon04

I cannot reproduce locally. So I'm groping in the dark. Maybe [o35389] helps.

[o35389] - see ​#josm16860 - Add ant resolve-tools target

[o35390] - see ​#josm16860 - Update to Apache Ivy 2.5.0

comment:80 Changed 11 days ago by simon04

I would like to get rid of source:trunk/tools/ivy/ivy.jar in the repository.

Variant 1: require Apache Ivy to be installed systemwide (as we do for Apache Ant)

Variant 2: download from https://josm.openstreetmap.de/nexus/content/repositories/public/ (as we do for plugins, as is shown in the most basic Ivy example)

  • build.xml

    diff --git a/build.xml b/build.xml
    index cc3a33e7a..11d2334c1 100644
    a b  
    1616         xmlns:unless="ant:unless"
    1717>
    1818    <target name="init-ivy">
     19        <property name="ivy.version" value="2.5.0"/>
    1920        <dirname property="base.dir" file="${ant.file.josm}"/>
    2021        <property name="lib.dir"   location="${base.dir}/lib"/>
    2122        <property name="tools.dir" location="${base.dir}/tools"/>
    2223        <property name="tools.ivy" location="${tools.dir}/ivy.xml"/>
    23         <taskdef resource="org/apache/ivy/ant/antlib.xml" uri="antlib:org.apache.ivy.ant" classpath="${tools.dir}/ivy/ivy.jar"/>
     24        <property name="ivy.jar.dir" location="${tools.dir}/ivy"/>
     25        <property name="ivy.jar.file" location="${ivy.jar.dir}/ivy-${ivy.version}.jar"/>
     26        <mkdir dir="${ivy.jar.dir}"/>
     27        <get src="https://josm.openstreetmap.de/nexus/content/repositories/public/org/apache/ivy/ivy/${ivy.version}/ivy-${ivy.version}.jar"
     28             dest="${ivy.jar.file}"
     29             skipexisting="true"
     30        />
     31        <echo>Apache Ivy ${ivy.jar.file}</echo>
     32        <taskdef resource="org/apache/ivy/ant/antlib.xml" uri="antlib:org.apache.ivy.ant" classpath="${ivy.jar.file}"/>
    2433    </target>
    2534    <target name="init-properties" depends="resolve">
    2635        <property environment="env"/>

Which variant should we take?

comment:81 Changed 11 days ago by Don-vip

I would say 2. Less disruptive for users. Probably as a new step that can be skipped for Debian build, or any system that provides ivy?
@sebastic your opinion?

comment:82 Changed 11 days ago by sebastic

No preference, the Debian package build won't call the ivy target.

comment:83 in reply to:  79 Changed 11 days ago by Don-vip

Replying to simon04:

I cannot reproduce locally. So I'm groping in the dark. Maybe [o35389] helps.

[o35389] - see ​#josm16860 - Add ant resolve-tools target

[o35390] - see ​#josm16860 - Update to Apache Ivy 2.5.0

It's... different :)

resolve-tools:

BUILD FAILED
/var/lib/jenkins/jobs/JOSM-Plugins/workspace/jdk/JDK8/build.xml:57: The following error occurred while executing this line:
/var/lib/jenkins/jobs/JOSM-Plugins/workspace/jdk/JDK8/build.xml:29: The following error occurred while executing this line:
/var/lib/jenkins/jobs/JOSM-Plugins/workspace/jdk/JDK8/build-common.xml:727: java.lang.NullPointerException
	at org.apache.ivy.ant.IvyAntSettings.getDefaultProperties(IvyAntSettings.java:322)
	at org.apache.ivy.ant.IvyAntSettings$1.execute(IvyAntSettings.java:268)
	at org.apache.ivy.ant.IvyAntSettings.createIvyEngine(IvyAntSettings.java:273)
	at org.apache.ivy.ant.IvyAntSettings.getDefaultInstance(IvyAntSettings.java:139)
	at org.apache.ivy.ant.IvyAntSettings.getDefaultInstance(IvyAntSettings.java:150)
	at org.apache.ivy.ant.IvyTask.getIvyInstance(IvyTask.java:77)
	at org.apache.ivy.ant.IvyTask.prepareTask(IvyTask.java:237)
	at org.apache.ivy.ant.IvyResolve.prepareTask(IvyResolve.java:236)
	at org.apache.ivy.ant.IvyTask.execute(IvyTask.java:258)
	at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:292)
	at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:99)
	at org.apache.tools.ant.Task.perform(Task.java:350)
	at org.apache.tools.ant.Target.execute(Target.java:449)

comment:84 Changed 11 days ago by Don-vip

The JDK11 build passed, I'm relaunching the JDK8 build.

comment:85 Changed 11 days ago by simon04

In 16183/josm:

see #16860 - <get> Apache Ivy, remove ivy.jar from repository

comment:86 in reply to:  49 Changed 11 days ago by simon04

Replying to simon04:

I've asked the apache-jcs maintainers to craft a new release which we could use via Apache Ivy: [JCS-204] Craft a new 2.3 release - https://issues.apache.org/jira/browse/JCS-204

Thomas Vandahl commented on JCS-204:

Unfortunately, the current trunk/master contains lots of breaking changes, so that a new release *must* be 3.x. This will require the package name as well as the Maven coordinates to reflect this. I'll see what I can do.

Update: I renamed the ticket to [JCS-204] Release JCS 3.0 - ASF JIRA

Last edited 10 days ago by simon04 (previous) (diff)

comment:87 Changed 11 days ago by simon04

@Vincent, how should we proceed with commons-jcs and jmapviewer?

Ad commons-jcs: could we publish the current SNAPSHOT to JOSM's Nexus and use 3.0-SNAPSHOT from there?

Ad jmapviewer: the latest release in JOSM's Nexus is 2.9, the latest released version 2.13 – Can we publish 2.10…2.13 to Nexus and include 2.13 via Apache Ivy

comment:88 in reply to:  87 ; Changed 11 days ago by Don-vip

Replying to simon04:

@Vincent, how should we proceed with commons-jcs and jmapviewer?

Ad commons-jcs: could we publish the current SNAPSHOT to JOSM's Nexus and use 3.0-SNAPSHOT from there?

Is 3.0-SNAPSHOT already available?

Ad jmapviewer: the latest release in JOSM's Nexus is 2.9, the latest released version 2.13 – Can we publish 2.10…2.13 to Nexus and include 2.13 via Apache Ivy

The jmapviewer nexus deploy currently relies on a very unreliable Jenkins job: https://josm.openstreetmap.de/jenkins/job/Nexus-JMapViewer/
We should make something most robust. Ideally from Ant? Something like an "ant release" target? For the old releases I can import them manually.

comment:89 in reply to:  88 Changed 11 days ago by Don-vip

Replying to Don-vip:

The jmapviewer nexus deploy currently relies on a very unreliable Jenkins job: https://josm.openstreetmap.de/jenkins/job/Nexus-JMapViewer/
We should make something most robust. Ideally from Ant? Something like an "ant release" target? For the old releases I can import them manually.

I have changed the Jenkins job to take VERSION as a parameter, seems to work. We can keep it is it until we switch to Gitlab I think. All versions should be available in Nexus now.

comment:90 in reply to:  88 ; Changed 10 days ago by simon04

Replying to Don-vip:

Is 3.0-SNAPSHOT already available?

3.0-SNAPSHOT is the version we are currently using via svn:externals, i.e., https://github.com/apache/commons-jcs/commit/366e4f001aaa72cf489f25d642ff88648a81a3fe
Can we compile and publish this version to our Nexus?

The jmapviewer nexus deploy currently relies on a very unreliable Jenkins job: https://josm.openstreetmap.de/jenkins/job/Nexus-JMapViewer/
We should make something most robust. Ideally from Ant? Something like an "ant release" target? For the old releases I can import them manually.

Thank you. Shall we migrate to jmapviewer 2.13 via Apache Ivy now?

Having accomplished those two steps would allow us to eliminate the svn:externals completely. :))

comment:91 in reply to:  90 Changed 10 days ago by Don-vip

Replying to simon04:

Can we compile and publish this version to our Nexus?

I setup an automatic build&deploy job: https://josm.openstreetmap.de/jenkins/job/Nexus-JCS/7/console
Can you please test?

Thank you. Shall we migrate to jmapviewer 2.13 via Apache Ivy now?

Sure!

Having accomplished those two steps would allow us to eliminate the svn:externals completely. :))

\o/

comment:92 Changed 10 days ago by simon04

In 16194/josm:

see #16860 - Resolve JMapViewer using Apache Ivy

comment:93 Changed 10 days ago by simon04

In 16195/josm:

see #16860 - Resolve commons-jcs-core using Apache Ivy

comment:94 Changed 10 days ago by simon04

Awesome, thank you! Seems to be working like a charm (at least on my system). In r16195 I had to remove the <exclude>s (i.e., not migrate them to extract-libraries) as otherwise spurious JCS/Cache exceptions would show up.

comment:95 Changed 10 days ago by Don-vip

Proguard fails.

comment:96 Changed 10 days ago by simon04

In 16197/josm:

see #16860 - Apache Ivy: exclude various commons-jcs classes from inclusion

comment:97 in reply to:  95 Changed 10 days ago by simon04

Replying to Don-vip:

Proguard fails.

The usual suspect ;)

comment:98 Changed 9 days ago by skyper

Is this ticket to blame that there was no latest released this morning ?

comment:99 in reply to:  98 Changed 8 days ago by anonymous

Replying to skyper:

Is this ticket to blame that there was no latest released this morning ?

no latest today, should be 16199.

comment:100 Changed 8 days ago by Don-vip

Ticket not to blame, random network error. Tomorrow it should work.

comment:101 Changed 4 days ago by simon04

In 16209/josm:

see #16860 - directory cleanup

comment:102 Changed 4 days ago by GerdP

After updating to r16211 I can no longer compile in Eclipse. I tried

ant clean dist

and a refresh in Eclipse but that did not help. I've also restarted Eclipse. I still have 1309 compile errors.

comment:103 Changed 4 days ago by GerdP

Seems all sources containing import org.openstreetmap.gui.* fail.

comment:104 Changed 4 days ago by GerdP

Fixed, had to do ivy resolve manually in Eclipse

comment:105 in reply to:  104 ; Changed 4 days ago by Bjoeni

Disclaimer: I didn't really follow / read all of this ticket and I never worked with Apache Ivy before (I only stumbled across this ticket because org.openstreetmap.gui.* failed to resolve).

But what's the best way to use JMapViewer in a JOSM plugin at the moment? I installed IvyDE in Eclipse and had it resolve, it added a reference to jmapviewer-2.13.jar. I then added that jar (from the Ivy cache directory) manually to the referenced libraries of my plugin. Is that the way to go as of right now or am I just not getting how Ivy is supposed to work?

comment:106 Changed 3 days ago by Klumbumbus

In 16219/josm:

see #16860, see #18880 - Adapt Netbeans config to r16165

comment:107 Changed 3 days ago by Klumbumbus

After r16166 Netbeans can't find spotbugs.jar anymore and the build fails, probably also due to r16194, as the message is:

C:\Users\stefa\Documents\OSM\josm\core\ide\netbeans\nbbuild.xml:90: C:\Users\stefa\Documents\OSM\josm\core\src\org\openstreetmap\gui\jmapviewer\images does not exist.
BUILD FAILED (total time: 1 minute 34 seconds)

(see also #18880)

comment:108 in reply to:  105 Changed 2 days ago by taylor.smock

Replying to Bjoeni:

Disclaimer: I didn't really follow / read all of this ticket and I never worked with Apache Ivy before (I only stumbled across this ticket because org.openstreetmap.gui.* failed to resolve).

But what's the best way to use JMapViewer in a JOSM plugin at the moment? I installed IvyDE in Eclipse and had it resolve, it added a reference to jmapviewer-2.13.jar. I then added that jar (from the Ivy cache directory) manually to the referenced libraries of my plugin. Is that the way to go as of right now or am I just not getting how Ivy is supposed to work?

I've had much the same issue. This (also) probably isn't the best way to go about it, but I've been changing the build path. So "Configure build path" -> "Order and Export" -> Check "Ivy". I've done this since most of my projects use JSON, and javax.json is now resolved by Ivy.

If you do that, you will have a diff for .classpath that should look like this:

  • .classpath

     
    3232                        <attribute name="test" value="true"/>
    3333                </attributes>
    3434        </classpathentry>
    35         <classpathentry kind="con" path="org.apache.ivyde.eclipse.cpcontainer.IVYDE_CONTAINER/?project=JOSM&amp;ivyXmlPath=ivy.xml&amp;confs=*&amp;ivySettingsPath=ivysettings.xml&amp;loadSettingsOnDemand=false&amp;ivyUserDir=&amp;propertyFiles="/>
     35        <classpathentry exported="true" kind="con" path="org.apache.ivyde.eclipse.cpcontainer.IVYDE_CONTAINER/?project=JOSM&amp;ivyXmlPath=ivy.xml&amp;confs=*&amp;ivySettingsPath=ivysettings.xml&amp;loadSettingsOnDemand=false&amp;ivyUserDir=&amp;propertyFiles="/>
    3636        <classpathentry kind="lib" path="test/lib/jfcunit.jar">
    3737                <attributes>
    3838                        <attribute name="test" value="true"/>

comment:109 Changed 29 hours ago by Don-vip

In 16221/josm:

see #16860 - remove copy of jmapviewer images in Netbeans project

comment:110 Changed 29 hours ago by Don-vip

@Klumbumbus can you please test Netbeans?

comment:111 Changed 23 hours ago by Klumbumbus

The build works again, but it still can't find spotbugs.jar.

comment:112 Changed 21 hours ago by Don-vip

In 16222/josm:

see #16860 - update spotbugs jar in Netbeans project

comment:113 Changed 21 hours ago by Don-vip

Done. Please check.

comment:114 Changed 21 hours ago by Klumbumbus

👍

comment:115 Changed 18 hours ago by Don-vip

Resolution: fixed
Status: newclosed

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The owner will remain team.
as The resolution will be set.
The resolution will be deleted.

Add Comment


E-mail address and name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.