Opened 3 years ago
Last modified 19 months ago
#15182 new enhancement
Standalone JOSM validator
Reported by: | Don-vip | Owned by: | team |
---|---|---|---|
Priority: | major | Milestone: | |
Component: | Core validator | Version: | |
Keywords: | modularization | Cc: |
Description (last modified by )
During Java 9 compatibility effort (#11924) I wondered if modularization could be applied to JOSM, which currently is a big spaghetti-based monolithic jar.
A clean modular architecture would be a huge task (don't know if it would really be possible at all, as we have a lot of interwoven dependencies) but I'd like to see if we can achieve a more affordable intermediate state, where we could build a standalone version of JOSM validator that could be run as follows:
java -jar josm-validator.jar data.osm -o warnings.xml
This jar having no dependency on AWT/Swing/JavaFX. This way it could be run for example by a Java 9 server VM which does not include the java.desktop module.
It would not change anything on JOSM editor except probably some API changes, but as little as possible.
#14704 is a prerequisite.
Attachments (1)
Change History (124)
comment:1 Changed 3 years ago by
comment:6 Changed 3 years ago by
Description: | modified (diff) |
---|
comment:8 Changed 3 years ago by
Hmm, wouldn't it be more "satisfactory" to first make JOSM callable as standalone validator at all before you make it fully encapsulated? Even with many useless internal dependencies that would be a useful application.
comment:21 Changed 3 years ago by
Description: | modified (diff) |
---|
comment:25 Changed 3 years ago by
Hmm, I don't think undoRedo is really a GUI element. I can think of possible projects using it without a gui, as it is the central comnand handling instance.
comment:26 Changed 3 years ago by
I can make it a singleton so it can be called outside of MainApplication.
comment:49 Changed 3 years ago by
Exciting to see all those changes towards a more modular structure! How are we doing with respect to the upcoming release, is everything considered stable?
comment:50 Changed 3 years ago by
I think so. It requires some plugins to be updated (I'm on it) but it's often just moving classes around and updating the imports. I have finished with deprecating stuff in Main, I have now a lot of code that compiles without the GUI :)
Changed 3 years ago by
Attachment: | validator-standalone.patch added |
---|
comment:63 Changed 3 years ago by
With current build.xml + this task:
<!-- ** build standalone validator jar. --> <target name="standalone-validator-jar" depends="clean,javacc,compile-cots,create-revision,check-schemas"> <delete dir="validator-src"/> <delete dir="validator-build"/> <mkdir dir="validator-src"/> <copy todir="validator-src" includeemptydirs="false"> <fileset dir="${src.dir}"> <include name="org/openstreetmap/josm/*.java"/> <include name="org/openstreetmap/josm/command/*.java"/> <include name="org/openstreetmap/josm/data/**"/> <exclude name="org/openstreetmap/josm/data/imagery/**"/> <include name="org/openstreetmap/josm/gui/mappaint/**"/> <include name="org/openstreetmap/josm/gui/progress/**"/> <exclude name="org/openstreetmap/josm/gui/progress/swing/**"/> <include name="org/openstreetmap/josm/gui/tagging/presets/**"/> <include name="org/openstreetmap/josm/io/**"/> <exclude name="org/openstreetmap/josm/io/imagery/**"/> <exclude name="org/openstreetmap/josm/io/remotecontrol/**"/> <exclude name="org/openstreetmap/josm/io/session/**"/> <include name="org/openstreetmap/josm/tools/**"/> </fileset> </copy> <patch patchfile="patches/validator-standalone.patch" strip="1" dir="validator-src" failonerror="true"/> <mkdir dir="validator-build"/> <javac compiler="${javac.compiler}" sourcepath="" srcdir="validator-src" classpath="${build.dir}" destdir="validator-build" target="${java.lang.version}" source="${java.lang.version}" debug="on" includeantruntime="false" createMissingPackageInfoClass="false" encoding="UTF-8"> <compilerclasspath> <pathelement location="${error_prone_ant.jar}"/> </compilerclasspath> <compilerarg value="-Xlint:cast"/> <compilerarg value="-Xlint:deprecation"/> <compilerarg value="-Xlint:dep-ann"/> <compilerarg value="-Xlint:divzero"/> <compilerarg value="-Xlint:empty"/> <compilerarg value="-Xlint:finally"/> <compilerarg value="-Xlint:overrides"/> <!--<compilerarg value="-Xlint:rawtypes"/>--> <compilerarg value="-Xlint:static"/> <compilerarg value="-Xlint:try"/> <compilerarg value="-Xlint:unchecked"/> <!-- Undocumented argument to ignore "Sun internal proprietary API" warning, see http://stackoverflow.com/a/13862308/2257172 --> <compilerarg value="-XDignore.symbol.file"/> <compilerarg value="-Xep:ReferenceEquality:OFF" compiler="com.google.errorprone.ErrorProneAntCompilerAdapter"/> <compilerarg value="-Xep:ImmutableEnumChecker:OFF" compiler="com.google.errorprone.ErrorProneAntCompilerAdapter"/> <compilerarg value="-Xep:FutureReturnValueIgnored:OFF" compiler="com.google.errorprone.ErrorProneAntCompilerAdapter"/> <compilerarg value="-Xep:FloatingPointLiteralPrecision:OFF" compiler="com.google.errorprone.ErrorProneAntCompilerAdapter"/> <compilerarg value="-Xep:ShortCircuitBoolean:OFF" compiler="com.google.errorprone.ErrorProneAntCompilerAdapter"/> <compilerarg value="-Xep:LiteralClassName:OFF" compiler="com.google.errorprone.ErrorProneAntCompilerAdapter"/> <compilerarg line="-Xmaxerrs 1000"/> </javac> <copy todir="validator-build" failonerror="no" includeemptydirs="no"> <fileset dir="resources"/> </copy> </target>
r12545 (current tested): 626 errors
r12695 (current trunk): 344 errors
The three main problems ahead are:
- code relying on layers
- mappaint code. we need the MapCSS code for MapCSS validator test. But we don't need all the renderers, etc.
- tagging presets, very GUI-oriented right now. We need them also for validator tests, but not the GUI components
From all errors here are the packages missing the most, by order of occurrence:
1 package org.openstreetmap.josm.actions.SplitWayAction does not exist 1 package org.openstreetmap.josm.gui.dialogs.relation does not exist 1 package org.openstreetmap.josm.gui.draw.MapViewPath does not exist 1 package org.openstreetmap.josm.gui.help does not exist 1 package org.openstreetmap.josm.gui.layer.markerlayer does not exist 1 package org.openstreetmap.josm.gui.layer.OsmDataLayer does not exist 1 package org.openstreetmap.josm.gui.oauth does not exist 1 package org.openstreetmap.josm.gui.preferences.display does not exist 1 package org.openstreetmap.josm.gui.preferences does not exist 1 package org.openstreetmap.josm.gui.preferences.server.ProxyPreferencesPanel does not exist 2 package org.openstreetmap.gui.jmapviewer does not exist 2 package org.openstreetmap.josm.actions.JoinAreasAction does not exist 2 package org.openstreetmap.josm.gui.bugreport does not exist 2 package org.openstreetmap.josm.gui.dialogs does not exist 2 package org.openstreetmap.josm.gui.dialogs.relation.sort does not exist 2 package org.openstreetmap.josm.gui.io does not exist 2 package org.openstreetmap.josm.plugins does not exist 4 package org.openstreetmap.josm.gui.preferences.projection does not exist 4 package org.openstreetmap.josm.gui.preferences.server does not exist 6 package org.openstreetmap.josm.gui.layer.MainLayerManager does not exist 8 package org.openstreetmap.josm.actions does not exist 8 package org.openstreetmap.josm.gui.MapViewState does not exist 8 package org.openstreetmap.josm.gui.tagging.ac does not exist 12 package org.openstreetmap.josm.gui.layer.LayerManager does not exist 12 package org.openstreetmap.josm.gui.util does not exist 15 package org.openstreetmap.josm.gui.widgets does not exist 16 package org.openstreetmap.josm.gui.draw does not exist 22 package org.openstreetmap.josm.gui.layer does not exist
and here are the files with the most unresolved imports:
1 \SVN\josm\core\validator-src\org\openstreetmap\josm\command\AddCommand.java 1 \SVN\josm\core\validator-src\org\openstreetmap\josm\command\AddPrimitivesCommand.java 1 \SVN\josm\core\validator-src\org\openstreetmap\josm\command\ChangeCommand.java 1 \SVN\josm\core\validator-src\org\openstreetmap\josm\command\PurgeCommand.java 1 \SVN\josm\core\validator-src\org\openstreetmap\josm\data\cache\JCSCachedTileLoaderJob.java 1 \SVN\josm\core\validator-src\org\openstreetmap\josm\data\cache\JCSCacheManager.java 1 \SVN\josm\core\validator-src\org\openstreetmap\josm\data\osm\ChangesetCache.java 1 \SVN\josm\core\validator-src\org\openstreetmap\josm\data\osm\DataSet.java 1 \SVN\josm\core\validator-src\org\openstreetmap\josm\data\osm\visitor\paint\MapRendererFactory.java 1 \SVN\josm\core\validator-src\org\openstreetmap\josm\data\osm\visitor\paint\OffsetIterator.java 1 \SVN\josm\core\validator-src\org\openstreetmap\josm\data\PreferencesUtils.java 1 \SVN\josm\core\validator-src\org\openstreetmap\josm\data\validation\tests\DuplicateNode.java 1 \SVN\josm\core\validator-src\org\openstreetmap\josm\data\validation\tests\TagChecker.java 1 \SVN\josm\core\validator-src\org\openstreetmap\josm\gui\mappaint\ElemStyles.java 1 \SVN\josm\core\validator-src\org\openstreetmap\josm\gui\mappaint\mapcss\ExpressionFactory.java 1 \SVN\josm\core\validator-src\org\openstreetmap\josm\gui\mappaint\styleelement\AreaIconElement.java 1 \SVN\josm\core\validator-src\org\openstreetmap\josm\gui\mappaint\styleelement\MapImage.java 1 \SVN\josm\core\validator-src\org\openstreetmap\josm\gui\mappaint\styleelement\Symbol.java 1 \SVN\josm\core\validator-src\org\openstreetmap\josm\gui\tagging\presets\items\Link.java 1 \SVN\josm\core\validator-src\org\openstreetmap\josm\gui\tagging\presets\TaggingPresetSearchAction.java 1 \SVN\josm\core\validator-src\org\openstreetmap\josm\gui\tagging\presets\TaggingPresetSearchPrimitiveDialog.java 1 \SVN\josm\core\validator-src\org\openstreetmap\josm\io\MultiFetchOverpassObjectReader.java 1 \SVN\josm\core\validator-src\org\openstreetmap\josm\io\MultiFetchServerObjectReader.java 1 \SVN\josm\core\validator-src\org\openstreetmap\josm\io\OsmConnection.java 1 \SVN\josm\core\validator-src\org\openstreetmap\josm\io\OsmWriter.java 1 \SVN\josm\core\validator-src\org\openstreetmap\josm\tools\bugreport\BugReport.java 1 \SVN\josm\core\validator-src\org\openstreetmap\josm\tools\bugreport\BugReportQueue.java 1 \SVN\josm\core\validator-src\org\openstreetmap\josm\tools\Geometry.java 1 \SVN\josm\core\validator-src\org\openstreetmap\josm\tools\ImageOverlay.java 1 \SVN\josm\core\validator-src\org\openstreetmap\josm\tools\ImageProvider.java 1 \SVN\josm\core\validator-src\org\openstreetmap\josm\tools\OsmUrlToBounds.java 1 \SVN\josm\core\validator-src\org\openstreetmap\josm\tools\PlatformHookWindows.java 1 \SVN\josm\core\validator-src\org\openstreetmap\josm\tools\Shortcut.java 2 \SVN\josm\core\validator-src\org\openstreetmap\josm\data\osm\event\DatasetEventManager.java 2 \SVN\josm\core\validator-src\org\openstreetmap\josm\data\osm\FilterModel.java 2 \SVN\josm\core\validator-src\org\openstreetmap\josm\data\osm\visitor\paint\AbstractMapRenderer.java 2 \SVN\josm\core\validator-src\org\openstreetmap\josm\data\osm\visitor\paint\ArrowPaintHelper.java 2 \SVN\josm\core\validator-src\org\openstreetmap\josm\data\projection\Projections.java 2 \SVN\josm\core\validator-src\org\openstreetmap\josm\data\validation\OsmValidator.java 2 \SVN\josm\core\validator-src\org\openstreetmap\josm\data\validation\PaintVisitor.java 2 \SVN\josm\core\validator-src\org\openstreetmap\josm\data\validation\tests\PublicTransportRouteTest.java 2 \SVN\josm\core\validator-src\org\openstreetmap\josm\gui\mappaint\styleelement\NodeElement.java 2 \SVN\josm\core\validator-src\org\openstreetmap\josm\gui\mappaint\styleelement\placement\CompletelyInsideAreaStrategy.java 2 \SVN\josm\core\validator-src\org\openstreetmap\josm\gui\mappaint\styleelement\placement\PartiallyInsideAreaStrategy.java 2 \SVN\josm\core\validator-src\org\openstreetmap\josm\gui\mappaint\styleelement\placement\PositionForAreaStrategy.java 2 \SVN\josm\core\validator-src\org\openstreetmap\josm\gui\tagging\presets\items\Check.java 2 \SVN\josm\core\validator-src\org\openstreetmap\josm\gui\tagging\presets\TaggingPresetSelector.java 2 \SVN\josm\core\validator-src\org\openstreetmap\josm\io\auth\AbstractCredentialsAgent.java 2 \SVN\josm\core\validator-src\org\openstreetmap\josm\io\auth\JosmPreferencesCredentialAgent.java 2 \SVN\josm\core\validator-src\org\openstreetmap\josm\io\DefaultProxySelector.java 2 \SVN\josm\core\validator-src\org\openstreetmap\josm\io\GeoJSONWriter.java 2 \SVN\josm\core\validator-src\org\openstreetmap\josm\io\MessageNotifier.java 2 \SVN\josm\core\validator-src\org\openstreetmap\josm\io\OsmApi.java 3 \SVN\josm\core\validator-src\org\openstreetmap\josm\command\Command.java 3 \SVN\josm\core\validator-src\org\openstreetmap\josm\data\osm\event\SelectionEventManager.java 3 \SVN\josm\core\validator-src\org\openstreetmap\josm\data\osm\visitor\paint\StyledMapRenderer.java 3 \SVN\josm\core\validator-src\org\openstreetmap\josm\data\osm\visitor\paint\WireframeMapRenderer.java 3 \SVN\josm\core\validator-src\org\openstreetmap\josm\gui\tagging\presets\TaggingPresetItem.java 3 \SVN\josm\core\validator-src\org\openstreetmap\josm\tools\bugreport\BugReportSender.java 3 \SVN\josm\core\validator-src\org\openstreetmap\josm\tools\RightAndLefthandTraffic.java 4 \SVN\josm\core\validator-src\org\openstreetmap\josm\data\osm\history\HistoryDataSet.java 4 \SVN\josm\core\validator-src\org\openstreetmap\josm\gui\mappaint\styleelement\placement\OnLineStrategy.java 4 \SVN\josm\core\validator-src\org\openstreetmap\josm\gui\tagging\presets\items\Combo.java 4 \SVN\josm\core\validator-src\org\openstreetmap\josm\gui\tagging\presets\items\Text.java 5 \SVN\josm\core\validator-src\org\openstreetmap\josm\command\DeleteCommand.java 5 \SVN\josm\core\validator-src\org\openstreetmap\josm\data\osm\visitor\paint\relations\MultipolygonCache.java 6 \SVN\josm\core\validator-src\org\openstreetmap\josm\gui\tagging\presets\TaggingPreset.java 7 \SVN\josm\core\validator-src\org\openstreetmap\josm\data\UndoRedoHandler.java 7 \SVN\josm\core\validator-src\org\openstreetmap\josm\gui\mappaint\MapPaintMenu.java
comment:65 Changed 3 years ago by
Replying to Don-vip:
During Java 9 compatibility effort (#11924) I wondered if modularization could be applied to JOSM, which currently is a big spaghetti-based monolithic jar.
A clean modular architecture would be a huge task (don't know if it would really be possible at all, as we have a lot of interwoven dependencies) but I'd like to see if we can achieve a more affordable intermediate state, where we could build a standalone version of JOSM validator that could be run as follows:
java -jar josm-validator.jar data.osm -o warnings.xml
This jar having no dependency on AWT/Swing/JavaFX. This way it could be run for example by a Java 9 server VM which does not include the java.desktop module.
I have similar long-term ideas, namely a stanalone MapCSS renderer and a projection library. Both should run in headless mode as well and come with command line arguments for basic use. After you've done the groundwork, it should be a piece of cake! ;)
comment:67 Changed 3 years ago by
When doing this, wouldn't it be wise to externalize that code completely?
It will make it much easier to re-use and normal contributors will be less likely to break things since the "normal" build will fail as well if the dependencies are not met.
I'd suggest the following for now:
- josm
- DataSet (+ all data set related classes)
- MapCSS
- DataSet
- MapPainting (Projection, MapView point conversion, MapViewPaintable)
- Imagery
- MapPainting
- Commands (that modify the data set)
- DataSet
Everything depends on data set for now.
For the preferences, I'd suggest to use a separate module for the storage classes. We can then use those preferences in the MapCSS/MapPainting/... modules and keep the user interface code for them in josm main for now.
comment:68 Changed 3 years ago by
Milestone: | → 17.09 |
---|
comment:69 Changed 3 years ago by
This doesn't work. Validator depends on MapCSS, but I don't want it to depend on MapView, which extends JComponent and imports a lot of awt/swing classes. I haven't looked exactly yet how to split the MapCSS parsing from the MapView/paint code.
The first step is to cut Swing out of our core data classes. That's I am trying to achieve. We'll see next how/if we can modularize further.
comment:70 Changed 3 years ago by
We can start with very informal modules, i.e. move some of the .java files into different directories. In terms of the build process, there is almost no change at all, we simply tell javac
to find the source files in multiple directories.
Then, gradually, the dependency structure can be carved out more clearly, eventually to be turned into proper Java 9 modules.
Here is a possible directory structure:
modules |- org.openstreetmap.josm.app |- src |- build.xml |- org.openstreetmap.josm.validator |- src |- main |- java |- module-info.java |- org/openstreetmap/josm/data/validation/*.java |- resources |- ignoretags.cfg |- deprecated.mapcss ... |- test |- java |- resources |- build.xml |- org.openstreetmap.josm.mapcss |- org.openstreetmap.josm.projection |- org.openstreetmap.jmapviewer ... build.xml README
The directory names (org.openstreetmap.josm.validator
, ...) are the names of later the Java 9 modules as per the module naming convention.
Switching to the standard Maven directory layout for each module (like in the draft) would be neither a big advantage nor disadvantage. On the plus side, it may be easier for new developers who are familiar with the Maven layout.
One question that remains, is if all this is worthwhile and achievable or a hopeless endeavor. ;)
comment:71 Changed 3 years ago by
That's about what I had in mind for the long term :)
It is worthwile. Once we have clean modules we can publish them to Maven Central. I'd like to see JOSM core classes become the standard OSM Java library :) There would be no need for other people to create their own (osm-common or osmapi for example)
comment:81 Changed 3 years ago by
r12750 is down to 282 errors. List of problematic files:
org/openstreetmap/josm/command/DeleteCommand.java org/openstreetmap/josm/data/cache/JCSCachedTileLoaderJob.java org/openstreetmap/josm/data/cache/JCSCacheManager.java org/openstreetmap/josm/data/osm/ChangesetCache.java org/openstreetmap/josm/data/osm/DataSet.java org/openstreetmap/josm/data/osm/event/DatasetEventManager.java org/openstreetmap/josm/data/osm/event/SelectionEventManager.java org/openstreetmap/josm/data/osm/FilterModel.java org/openstreetmap/josm/data/osm/history/HistoryDataSet.java org/openstreetmap/josm/data/osm/visitor/paint/AbstractMapRenderer.java org/openstreetmap/josm/data/osm/visitor/paint/ArrowPaintHelper.java org/openstreetmap/josm/data/osm/visitor/paint/MapRendererFactory.java org/openstreetmap/josm/data/osm/visitor/paint/OffsetIterator.java org/openstreetmap/josm/data/osm/visitor/paint/relations/MultipolygonCache.java org/openstreetmap/josm/data/osm/visitor/paint/StyledMapRenderer.java org/openstreetmap/josm/data/osm/visitor/paint/WireframeMapRenderer.java org/openstreetmap/josm/data/PreferencesUtils.java org/openstreetmap/josm/data/projection/Projections.java org/openstreetmap/josm/data/validation/OsmValidator.java org/openstreetmap/josm/data/validation/PaintVisitor.java org/openstreetmap/josm/data/validation/tests/DuplicateNode.java org/openstreetmap/josm/data/validation/tests/TagChecker.java org/openstreetmap/josm/gui/mappaint/ElemStyles.java org/openstreetmap/josm/gui/mappaint/mapcss/ExpressionFactory.java org/openstreetmap/josm/gui/mappaint/MapPaintMenu.java org/openstreetmap/josm/gui/mappaint/styleelement/AreaIconElement.java org/openstreetmap/josm/gui/mappaint/styleelement/MapImage.java org/openstreetmap/josm/gui/mappaint/styleelement/NodeElement.java org/openstreetmap/josm/gui/mappaint/styleelement/placement/CompletelyInsideAreaStrategy.java org/openstreetmap/josm/gui/mappaint/styleelement/placement/OnLineStrategy.java org/openstreetmap/josm/gui/mappaint/styleelement/placement/PartiallyInsideAreaStrategy.java org/openstreetmap/josm/gui/mappaint/styleelement/placement/PositionForAreaStrategy.java org/openstreetmap/josm/gui/mappaint/styleelement/Symbol.java org/openstreetmap/josm/gui/tagging/presets/items/Check.java org/openstreetmap/josm/gui/tagging/presets/items/Combo.java org/openstreetmap/josm/gui/tagging/presets/items/Link.java org/openstreetmap/josm/gui/tagging/presets/items/Text.java org/openstreetmap/josm/gui/tagging/presets/TaggingPresetItem.java org/openstreetmap/josm/gui/tagging/presets/TaggingPreset.java org/openstreetmap/josm/gui/tagging/presets/TaggingPresetSearchAction.java org/openstreetmap/josm/gui/tagging/presets/TaggingPresetSearchPrimitiveDialog.java org/openstreetmap/josm/gui/tagging/presets/TaggingPresetSelector.java org/openstreetmap/josm/io/auth/AbstractCredentialsAgent.java org/openstreetmap/josm/io/auth/JosmPreferencesCredentialAgent.java org/openstreetmap/josm/io/DefaultProxySelector.java org/openstreetmap/josm/io/GeoJSONWriter.java org/openstreetmap/josm/io/MessageNotifier.java org/openstreetmap/josm/io/MultiFetchOverpassObjectReader.java org/openstreetmap/josm/io/MultiFetchServerObjectReader.java org/openstreetmap/josm/io/OsmApi.java org/openstreetmap/josm/io/OsmConnection.java org/openstreetmap/josm/io/OsmWriter.java org/openstreetmap/josm/tools/bugreport/BugReport.java org/openstreetmap/josm/tools/bugreport/BugReportQueue.java org/openstreetmap/josm/tools/bugreport/BugReportSender.java org/openstreetmap/josm/tools/ImageOverlay.java org/openstreetmap/josm/tools/ImageProvider.java org/openstreetmap/josm/tools/OsmUrlToBounds.java org/openstreetmap/josm/tools/RightAndLefthandTraffic.java
comment:120 Changed 3 years ago by
Milestone: | 17.09 → 17.10 |
---|
comment:123 Changed 19 months ago by
Owner: | changed from Don-vip to team |
---|---|
Status: | assigned → new |
In 12620/josm: