Infrastructure migration

[michael2402]

Meta-Ticket to coordinate the timeline of infrastructure conversion (build system, ci, GIT, dependencies, ...)

I (michael2402) will work on creating some more smaller tickets and linking all the tickets here as soon as Jenkins is done.

---

Legend: yellow=todo, green=done, blue=optional

Mermaid code for generating above graph using ​https://mermaid-js.github.io/mermaid-live-editor/

graph TD
gitlab[setup GitLab server]:::todo
gitlabCi[setup GitLab CI or Jenkins for Git]:::todo
ciArtifcat[use CI to create artefacts]:::done
git[move to Git]:::todo

gradle[move to Gradle]:::todo
dropAnt[drop Ant and Eclipse]:::todo
gradleSubproject[move to Gradle subprojects]:::todo


gitSvgSalamander[clone svgSalamander to Git]:::optional
patchSvgSalamander[patch svgSalamander with our changes]:::optional
ciArtifcatSvgSalamander[use CI to create svgSalamander artifacts]:::optional
ivySvgSalamander[use svgSalamander via Ivy]:::optional

ivyJMapViewer[use JMapViewer via Ivy]:::done
gitJMapViewer[move JMapViewer to Git]:::todo
gradleJMapViewer[move JMapViewer to Gradle]:::todo
ivyApacheCommons[use Apache Commons via Ivy]:::done
ivyTools[use compile/test tools via Ivy]:::done

gitSvgSalamander --> patchSvgSalamander --> ciArtifcatSvgSalamander --> ivySvgSalamander --> gradle
ciArtifcat --> ciArtifcatSvgSalamander
gitlab --> gitlabCi --> ciArtifcat --> gradle
gitlab --> gitJMapViewer
gitlab --> git --> gradle --> dropAnt --> gradleSubproject
ivyJMapViewer --> gitJMapViewer --> gradleJMapViewer --> ciArtifcat
ivyJMapViewer --> ivyApacheCommons --> ivyTools --> git

dropGetopt[drop getopt]:::done

classDef todo fill:#ffd92f;
classDef done fill:#a6d854;

[Don-vip]

Thanks Michael for this graph! :)

[simon04]

I've updated the dependency graph using mermaid.

For now, we're pretty good on the Java part (Ant, Ivy, Maven). The next steps would involve the Git part.

[GerdP]

What is meant by dropping Eclipse?

[simon04]

I guess (not written by me) removing source:trunk/.settings since that can be generated automatically from the Gradle configuration, see #8269.

[Don-vip]

The "move to Gradle" part won't happen as such. I'm ok to add Gradle support for those who want to use it, but I really don't like it, and prefer Maven by far. I would rather see a common build.properties shared between Ant/Ivy, Maven and Gradle builds. I talked to some Linux package maintainers and heard similar feedback: Gradle is not their favorite piece of meat.

[simon04]

@team: I would like to define a master plan for the infrastructure migration and find a concerted setup which can realistically be achieved within the next six months (not necessarily the one from comment:description).

After having defined the desired setup, we can define individual tasks and bring them in a sensible order. Ideally, we could tick off at least one of those tasks with our monthly JOSM releases and move with a steady pace towards our goal.

What do you think?

As of today, 2020-11-25, our infrastructure consists of (no worries, I don't want to change everything):

	JOSM core	JOSM plugins
server infrastructure	Hetzner sponsored by FOSSGIS
website	Trac	Trac or GitHub or GitLab
project management	Trac	Trac or GitHub or GitLab
version control system	SVN (+ GitHub mirror)	SVN or Git
build system	Apache Ant	Apache Ant and/or Gradle
dependency management	Apache Ivy	.jar files or Apache Ivy or Gradle
continuous integration	Jenkins + GitHub Actions + AppVeyor	Jenkins or GitHub Actions or GitLab CI
crafting releases	JOSM server script + GitHub Actions (for macOS)	manually or GitHub Actions or GitLab CI
publishing releases	download	svn commit .jar to SVN or GitHub releases or GitLab releases
internationalization	Launchpad	Launchpad or Transifex

[michael2402]

The problem I see is that we should attempt to keep the infrastructure simple instead of adding more complexity. Currently, a lot of workflows depend on the current setup.

Therefore, we should define what we want to support in the future. And I would really like to drop things on the way. For some things, it is much easier to not do a step-by-step migration. E.g. when changing the build system + deployment, we change both at the same time. When moving to git, we need to move to a new versioning scheme. List so far:

Versioning

• A latest version (=> How to tag? How to define versions there? Automate daily?)
• A tested version (tagged monthly)

Build environments

• Development with Eclipse (?)
• Development with IntelliJ

Deployments:

• Download (latest + tested)
• Ubuntu packages
• Webstart

So this is what I propose to get everything up and running without having a lot of compatibility-pain during migration (I learned some things form several to-gitlab/to-jenkins/to-github migrations in the last years and did both 'smooth' transitions and complete changes…):

• Set up a gitlab server + the gitlab ci runner.
• Export JSON SVN to git repository (a first test run) and set it up on gitlab.
• In Git: Change the build system to gradle
• Set up deployment of all artifacts in a single point (e.g. always nexus)
• Now set up the ci tasks
• * A tasks that builds JOSM
• * A nighty scheduled task that builds JOSM main branch and deploys that artifact as <timestamp> version (we could also do this on every successful build)
• * A task that is run on every git tag and that deploys the artifact as tested with that tag name (we then only push tags in the form of <year>.<month>.0)
• Create a wiki page with download links to nexus (for testing)
• Now we can do the adjustments to JOSM updater, set up the webstart for this, … - but no real development in the git repo!
• Do the diff of what we needed to change in the repo to make everything work (gitlab pipelines are stored in the project, so they are included as well)
• Drop that git repo
• Set SVN to read only
• Re-Do the export, for real this time.
• Apply our diff
• Migrate plugins (the ones we won't drop), one by one
• Put download links to new JOSM on main page

That way, we can do the development on the new infrastructure without breaking the working JOSM versions.

The man problem with the version I see are the plugins. Therefore, I would suggest to move every plugin to GIT as well. And then just use the new versions as min/max compatibility. Here we can use the advantage of building everything new: We don't have to support any migration. The new plugin versions will work on new JOSM. Old plugin versions will work on old JOSM. We just set up the new infrastructure, switch JOSM development to it at one point, migrate the most used plugins, but old JOSM still works. So if anyone needs a non-migrated plugin, they can stick to old JOSM for some more months.

At some day in the future, we might then shut down the old system.

[stoecker]

Like I asked multiple times in the past. BEFORE discussing any infrastructure changes explain the BENEFITS.

We have a well working system where several man-years development went into. What are the problems you want to solve and what gets better when we change infrastructure?

[stoecker]

BTW: What means "continuous integration" "Jenkins + GitHub Actions + AppVeyor" for core. Only Jenkins is offical.

[michael2402]

Replying to stoecker:

Like I asked multiple times in the past. BEFORE discussing any infrastructure changes explain the BENEFITS.

We had this discussion multiple times. I am tired of it and that is the reason why I did not work on it any more in the past years. We had a completely working gradle build (that is now broken again) and would save us the hassle of maintaining two build systems (ant+eclipse) and keep them in sync plus allow intelliJ-users and users of other IDEs to use their IDE for developing. And after that, I lost interest in developing for the trash can.

Replying to stoecker:

We have a well working system where several man-years development went into.

And this is the reason to change it. JOSM is a relatively small project: Not much integration in external API, no external workflows, just a jar (and some more for the plugins). You should not need several man-years to develop a build+update system for it. I maintain some more complex projects and for each of them, much less work is spent on such tasks.

The whole JOSM infrastructure has just grown over the years, is hard to maintain and we will spend several more man-years on it.

Replying to stoecker:

Only Jenkins is offical.

Official is a very vague thing in open source.

We have plugins that use Github, we have some that use gitlab ci, we have some that use SVN + Jenkins.

[simon04]

@stocker: If the current infrastructure should not be changed, please close all tickets suggesting a change as WONTFIX, and do not waste contributor's time.

@stocker: I suggested to find a infrastructure setup that is accepted by all JOSM core team members.

[stoecker]

Yes, we had this discussion multiple times and at the end a clear benefit of proposed changes never was there.

• We don't have two built systems. JOSM is built with ant. Plugins are under the control of the plugin authors. If they don't follow josm core it is the authors decision.
• Plugins also in future will use whatever version control system or build system or translation system the author chooses. That's why they are plugins.
• You seem to have no idea what josm server really does. The server infrastructure provides a lot of stuff which is essential to the whole ecosystem. You don't see this, so you think its not worth anything. The trac integration of josm handles a large amount of things for everyday work of mappers. That's a whole own application which you totally overlook it seems.
• The JOSM infrastructure is not hard to maintain. It's actually pretty straightforward. We can setup the server and all the services from scratch with a few commands which are scripted. A bit complicated is only that Jenkins stuff mainly because of the missing separation between working data and configuration. And it seems to break easily when installing updates. I really would appreciate to have a store config and resetup jenkins script.
• Wiki, JOSM-Trac services, Tickets, Jenkins all run smoothly for years now and get used intensively. We're releasing a new release nearly every month and it has many changes included.

The only infrastructure thing we actually agreed on was to move the plugins to a GitLab instance. If that would have been a big success a migration of core could follow. That task is stalled now for so long that I actually don't see any sense in it anymore. But Vincent has to decide if he wants to continue with this or not.

Even the Transifex migration makes no sense anymore as Launchpad got fixed and Transifex got worse. I'd prefer a JOSM server based solution, but there's isn't a software for this available, so we have to use an external service.

Regarding many of the changes tested in plugins:

• I don't see advantages of gradle over ant in plugins. It's another build system, but for me there are more disadvantages than benefits.
• Transifex translation don't have better quality than Launchpad based ones
• Now that lots of plugins migrated to GitHub and I watched this for some years I don't see any of the benefits which have been told:
  • There aren't more developers
  • Plugins are abandoned with the same rate as in SVN
  • There aren't any more pull requests from outsiders than patches via tickets or outsider SVN commits.
  • Not using GitHub, but GitLab will not make that better, as the GitHub UI is much better than the competitors
  • But it is more complex for me or Vincent to update and build the abandoned ones compared to the SVN before

I'm not opposed to changes which make sense and even if I'm not really happy with the ivy stuff I can see the benefits. But from the very beginning I communicated that:

• Changes must not replace internal services with external services which cannot be replaced
• There must be benefits beside "it's new"

So yes, if you can't name benefits except unifying the core to the current ideas of plugin authors, then close the tickets.

[simon04]

This biggest drawbacks of the current setup from my point of view are:

• Reviewing patches is hard: no continuous integration, patch files are unhandy (download/apply/cleanup), hard to give feedback to a specific location in the code, hard to follow N revisions of the patch – the GitHub pull request (or GitLab merge request) workflow is way easier, more powerful, and provides ​good tooling.
• Working on larger changes is hard: mostly the same as above.
• Working with plugins is very cumbersome at the moment: every* plugin is hosted somewhere else, every* plugin requires different IDE setup, every* plugin has a different commit/merge workflow, every* plugin has to be released differently (every* has to be taken with a grain of salt).
• The GitHub mirroring script I've been running since 2015 has been broken too often.

In addition, I think it's hard to get started for newcomers / casual contributors.

Personally, I've been using git svn for JOSM development since 2012, and JOSM is the only project requirering me to install Subversion from all projects I contribute to.

I don't suggest to change everything, but I hope to see the mentioned drawbacks vanish eventually in 2021…

[Don-vip]

My cents:

• Ant build works, doesn't need a lot of work to maintain and is being used by many outside of us (linux maintainers etc.) so I agree with Dirk to keep it
• On the other hand, let's face it: ant itself is not properly maintained, as very few projects still use it. It's a shame to see major regressions in JUnit reports ​not being addressed. For this reason I think it would be good to have the possibility to additionally build JOSM using an up-to-date tool such as Maven (I don't like Gradle). This way we could restore the reporting mechanisms I setup in Jenkins long ago, that worked fine with JUnit 4+Ant junit task, and don't work anymore with JUnit 5 + Ant junitlauncher task, so that I had to commit an ugly workaround. But this is low priority.
• I agree with everyone except Dirk on the need to switch to git over SVN. git is way faster and allow everyone to maintain their own branches without having to deal with patches. The patch approach is error-prone, difficult to read, and very time consuming (I often need to ask for new patch files that don't apply nicely, it seems tools behavirour vary between IDEs, OS, and so on). Like a lot of you, I use git daily at work and for other open source projects and SVN for JOSM is more and more a pain to use. This is one aspect of JOSM development that is not fun at all. This is not the main one, but still one of the reasons I lost interest in JOSM development this year.
• I enjoyed Simon's and Taylor's efforts to switch to Ivy + JUnit 5. As well as Stefan's for the SVG icons. This is great work and we should keep this spirit.
• I agree with Dirk for Transifex. The only reason I supported the move was the terrible performance of Launchpad, but this is issue is gone. So we can stay there.
• I really want to setup GitLab but I wanted to keep a single login/password for all of our tools. I couldn't find a way for GitLab so that's why I didn't work on this for a long time. Maybe we can simply create dedicated GitLab accounts. I find it less elegant but can't think of a better solution.
• I'd like to use GitLab only for merge requests + GitLab CI/CD to validate that merge requests are ok. Nothing else (BTW I love Trac history view, I find it far superior to GitLab and GitHub UI).
• Jenkins is kinda broken because I didn't maintain it this year, that's all. I'm looking at it some times to times, and restoring things one by one.

The target I'd like to see:

• switch everything to git. Keep current JOSM revision number mechanism by simply counting revisions on "master" branch. Do the same for plugins.
• keep Trac for everything, except we ask people to use the tickets only for bug reports and feature requests. Patches disappear in favour to GitLab merge requests. This requires us, plugin developers, external contributors, to create a "JOSM developer account" on GitLab which unfortunately cannot be the same as Trac.
• ask every plugin author to switch to the official JOSM GitLab.
• keep Jenkins current jobs, as it performs a better job than GitLab on reporting things
• introduce GitLab CI only for merge requests, to check JUnit breakages, code style, etc.
• Configure SonarQube to report issues on merge requests (this is very easy I do it at work)
• keep GitHub actions for tasks we cannot do on the JOSM server, such as performing native compilations/packaging that require a macOS or Windows runner, which we can get for free with GitHub.

[michael2402]

OK, Track just deleted my comment on @stoecker because I tried to respond to two comments at once. But basically, it was nothing new, just the same arguments as we had last time. If you want to stick with "works for me", then don't complain if nobody else is doing work. I won't do a JOSM core GSoC, just because sending around patches is a pain. I will happily do an other PT assistant or other project.

[GerdP]

I feel a bit wrong in this discussion because I have almost no experience with the proposed replacements for tools nor do I know much about those that we actually use. Most of the problems that I have are in Eclipse, so I should probably try IntelliJ if that doesn't show them:

• I still have to run checkstyle manually because Eclipse plugin doesn't work with our config. This is just boring extra work.
• Similar problems with the combination of ivy and svn plugin: Sometimes I have to click on ivy resolve to make things work again after an svn update on the command line. Might be caused by my preference "Set file dates to the “last commit time”, something that I was missing in git.
• Debugging plugins rarely allows "hot code replacement" or whatever it is called, so a change in the plugin source typically means to stop JOSM; do ant clean dist for the plugin, manual copy of <plugin>.jar, start JOSM

I use TortoiseSVN for forensic actions like diff, blame, log, but also command line for svn update, patch, revert. I guess I can learn to use git if that is the favorite for so many others, and maybe TortoiseGit works better now. The last Time that I tried it is probably two years ago and it messed up TortoseSVN which I still want to use for my other project mkgmap. One of my problems with git was related to eol differences when using patches with either Unix or Windows style eol, but I don't remember the details.
So, if you manage to migrate to GitLab or GitHub I'll just try to clone it(or whatever I have to do then) and learn to work around the problems in that tool.

[stoecker]

• I agree with everyone except Dirk on the need to switch to git over SVN. git is way faster and allow everyone to maintain their own branches without having to deal with patches. The patch approach is error-prone, difficult to read, and very time consuming (I often need to ask for new patch files that don't apply nicely, it seems tools behavirour vary between IDEs, OS, and so on).

I also find patch handling/submit requests of Git(hub) better than patch files (OTOH when properly using a step by step approach patches aren't worse, only these large unplanned patchset are trouble), but I still don't believe that at the end the small benefits will be worth the efforts spent. But I'm open to be convinced.

Like a lot of you, I use git daily at work and for other open source projects and SVN for JOSM is more and more a pain to use.

Well, I use SVN at work and using git is often a pain in the ass for normally extremely easy tasks. But I use for each project the tools of that project (svn, git, osc, bzr, ...) and I really can't understand the argument that the version control software is sooo important. I'd understand that if we would use RCS, SCCS, CVS or something alike that you would say "What's that", but SVN is a modern easy usable software and JOSM has unlike the linux kernel a small team and no need for distributed development.

This is one aspect of JOSM development that is not fun at all. This is not the main one, but still one of the reasons I lost interest in JOSM development this year.

Well, loosing interest after some time is pretty normal and I wondered how you could keep up with the workload you did for JOSM.

• I really want to setup GitLab but I wanted to keep a single login/password for all of our tools. I couldn't find a way for GitLab so that's why I didn't work on this for a long time. Maybe we can simply create dedicated GitLab accounts. I find it less elegant but can't think of a better solution.
• I'd like to use GitLab only for merge requests + GitLab CI/CD to validate that merge requests are ok. Nothing else (BTW I love Trac history view, I find it far superior to GitLab and GitHub UI).

I understand the wish for this and that's also the reason why I didn't said no in the beginning. I don't find the move to Git useful, but I'm open to be convinced.

The target I'd like to see:

• switch everything to git. Keep current JOSM revision number mechanism by simply counting revisions on "master" branch. Do the same for plugins.

I'm also open to this for the core, but only if, like discussed, we test it on the plugins and new procedure works there reliable.

• ask every plugin author to switch to the official JOSM GitLab.

Forget it :-)

[Stereo]

One big advantage of git is that it is, in 2020, well known and widely used. As an open source project, it's important to make the barrier to entry easy, and remove the not fun stuff from the workflows because otherwise existing contributors will lose interest. As someone who's not really part of the team, the barrier to contributing to JOSM is currently huge, and the environment could be more inclusive.

Migrating from trac and svn to git and gitlab or github is always a project that is long and not fun, and we're all basically just asking @stoecker to do it for free when he doesn't really see the benefits for his workflow. I understand his lack of enthusiasm. On the other hand, I don't sense an opposition from him if someone else were to do the work. (Dirk, I don't want to speak for you or put words in your mouth!)

On top of the work that an usual migration represents, there's about 3000 lines of Python 2 running essential stuff like wiki macros, all the endpoints like the imagery xml generation. There's build scripts in Perl, Makefiles that might or might not depend on trac, and I'm not sure what else there is. The first thing I would recommend is better documentation of what's happening behind the curtain. The stuff that can be shared should be in /scripts, and everything should in any case be documented on a wiki page.

The OSMF is supporting the development of the iD editor financially, and could support JOSM too. If there's been no one volunteering in two years to do the grunt work, we could look at hiring someone for the project.

[stoecker]

Replying to Stereo:

One big advantage of git is that it is, in 2020, well known and widely used. As an open source project, it's important to make the barrier to entry easy, and remove the not fun stuff from the workflows because otherwise existing contributors will lose interest. As someone who's not really part of the team, the barrier to contributing to JOSM is currently huge, and the environment could be more inclusive.

You know I'm in this business for more than two decades. Barrier to contribute to JOSM are actually very low (if you don't count the complexity to get into the code itself). For others you need to sign papers, transfer copyrights and often even for a bug report you need to do complex login procedures. JOSM is extremely mature, so patch contributions are the minority, most feedback are bug reports which usually get into the finer details instead of bigger problems. The git migration is no benefit for new contributors - it's an improvement (or not, depends on the view) for the core team.

Migrating from trac and svn to git and gitlab or github is always a project that is long and not fun, and we're all basically just asking @stoecker to do it for free when he doesn't really see the benefits for his workflow. I understand his lack of enthusiasm. On the other hand, I don't sense an opposition from him if someone else were to do the work. (Dirk, I don't want to speak for you or put words in your mouth!)

That's one important point. At the end Vincent and I have to continue maintaining whatever change we do. If Vincent wouldn't support the Git migration that would mean that I would be totally opposed to that. As Vincent also want this I actually agree to go that way.

On top of the work that an usual migration represents, there's about 3000 lines of Python 2 running essential stuff like wiki macros, all the endpoints like the imagery xml generation. There's build scripts in Perl, Makefiles that might or might not depend on trac, and I'm not sure what else there is. The first thing I would recommend is better documentation of what's happening behind the curtain. The stuff that can be shared should be in /scripts, and everything should in any case be documented on a wiki page.

Releasing this code under an OpenSource license does not really bring benefits and currently I'm opposed to this for a few reasons:

• Contributing to this is usually not possible, so we have no gain from releasing
• While I try to be security aware in development making it open source makes it easier to find potential flaws and any flaw in the server code makes an attack on the server possible
• It makes a big difference if I see thousands attempts to find a security hole in the Apache logs or if someone uses one single crafted attack based on the sources

The OSMF is supporting the development of the iD editor financially, and could support JOSM too. If there's been no one volunteering in two years to do the grunt work, we could look at hiring someone for the project.

In the years ago I thought about this and my answer to this is simple: If we start to pay people for doing JOSM work then either do it fully or not at all. I wouldn't contribute my free time to a project like JOSM when somebody else gets payed for doing that work.

There was one experience with paid work (turn-restrictions plugin) and actually that was not positive.

[michael2402]

Replying to GerdP:

• Debugging plugins rarely allows "hot code replacement" or whatever it is called, so a change in the plugin source typically means to stop JOSM; do ant clean dist for the plugin, manual copy of <plugin>.jar, start JOSM

This won't be solved with the new system. The problem is that hot replacement of classes only works in few occasions (e.g. if method signatures don't change or in same cases if a class is currently not in use). In several cases, you have to restart JOSM.

For plugin development, you can already use IntelliJ. Just place the gradle build script in your plugins root directory. You can find an example here: ​https://github.com/JOSM/pt_assistant/blob/master/build.gradle.kts Contact me if you have any problems with it, we can do it together. Then open the project in IntelliJ and everything is imported automatically. You don't need to clean then, because gradle supports real incremental builds.

I use TortoiseSVN for forensic actions like diff, blame, log, but also command line for svn update, patch, revert. I guess I can learn to use git if that is the favorite for so many others, and maybe TortoiseGit works better now

Don't use it. If you use IntelliJ, you have git built in. If you use eclipse, add the git plugin. VS code has good support as well, especially when merging. It integrates in the IDE, shows you git blame + current changes directly in the file, current branch in the corner of the window and you can commit directly form the IDE.

[GerdP]

I use TortoiseSVN because it integrates well into my most important tool, the Total Commander (a Norton Clone). If IntelliJ doesn't help much with plugins I have no motive to change the IDE, esp. not when it is not for free.

[michael2402]

Replying to GerdP:

I use TortoiseSVN because it integrates well into my most important tool, the Total Commander (a Norton Clone). If IntelliJ doesn't help much with plugins I have no motive to change the IDE, esp. not when it is not for free.

I don't see the need for total commander while developing plugins.

IntelliJ currently only helps with plugins. PT Assistant is developed using IntelliJ. It is free, just some plugins / modules cost money - none of which we need for JOSM plugin development.

For JOSM Core, it is a pain to set up because it uses ANT and a very complicated project setup with complex excludes, which is not documented but you need to set it up according to the eclipse or ant build files.

[mdk]

Replying to GerdP:

I use TortoiseSVN because it integrates well into my most important tool, the Total Commander (a Norton Clone). If IntelliJ doesn't help much with plugins I have no motive to change the IDE, esp. not when it is not for free.

Git is very well integrated into Eclipse. But I prefer "Git Extensions" on Windows, which has a very good UI for Git. Also "Git Extensions" integrate Git into the Windows Explorer.

[Stereo]

You know I'm in this business for more than two decades. Barrier to contribute to JOSM are actually very low (if you don't count the complexity to get into the code itself).

Everything is relative. Compared to a github project where I can create a PR in a few clicks without even having to create a local clone or talk to anyone, it's a bit heavy. You're on the inside, but from the outside, contributing to JOSM is intimidating and not easy to figure out.

[stoecker]

Replying to Stereo:

You know I'm in this business for more than two decades. Barrier to contribute to JOSM are actually very low (if you don't count the complexity to get into the code itself).

Everything is relative. Compared to a github project where I can create a PR in a few clicks without even having to create a local clone or talk to anyone, it's a bit heavy.

SVN checkout, changing code, svn diff and append to ticket isn't more work. With the "PR in a few clicks" you forget the git part before. And not to forget there are people out there who aren't happy that git requires you to download gigabytes of old code only to get started over a small network line. I understand that you guys learned with git and are used to it so much that you think it is the ultimate solution. When the JOSM developers prefer git that's a reason to consider switching.

But it is simply not true that git is superior to the workflow of JOSM for the usecase of JOSM.

I did software development when you had to send patches by mail (on disk) and had to wait 2 to 4 weeks for feedback. That was awful, but it was the way it worked. I remember the major step, when I actually could talk by email to somebody and the speedup that caused. If people cannot be motivated to do the very few necessary steps for contribution to JOSM then I have no interest in these people contributing. For sure they are that type which vanish as soon as the own patches starts to cause trouble which needs real debugging to find the problem.

You're on the inside, but from the outside, contributing to JOSM is intimidating and not easy to figure out.

Then try e.g. contributing to KDE, Linux kernel, Apache projects, GNU and so on. Not everything is a click-and-go and soon bitrotting GitHub project. I contributed and still do to a really large amount of projects. JOSM isn't a small utility, but a really large application and that in itself isn't easy.

[Stereo]

I'm slightly flattered that you think I'm too young to ever have accidentally broken stuff with cvs :).

I'm not forgetting the git part and the gigabytes: click on the little pencil at the top right of the code on, say, ​https://github.com/grischard/josm/blob/master/resources/styles/standard/elemstyles.mapcss and it will automatically create a fork, open it in an editor, walk you through creating a PR when you hit save at the bottom, and you will never have to download anything.

[GerdP]

Means it allows to produce a completely untested fork, right?

[Stereo]

Yeah, but then the CI tools automatically run on the pull request, like at the bottom of ​https://github.com/openstreetmap/chef/pull/362

(Don't look at the openstreetmap/josm PRs for this, it wasn't set up to run tests since we don't use PRs in our workflow.)

</tangent>

[GerdP]

BTW: Jenkins seems to have a problem.

[michael2402]

Replying to GerdP:

Means it allows to produce a completely untested fork, right?

I use github in some production projects and gitlab in some others. For each branch / PR, we even create a full release jar (and since it is a web application, we deploy it to a test instance named after the branch, so you can access it using your-branch-name.mytestserver.de). The cool thing for me as a mentor there (I rarely code, but do a lot of reviews) is that I can see the changes without the need to check out the project. Same goes for sonar: It runs on all PRs before they are merged. So I just need to review the code, write my comments in there (each can be marked as resolved on it's own), if I am happy I approve and the PR gets merged in master. Except for the developer originally developing the code, nobody needs to even check out the project and we still have all changes reviewed and run all automated tests on them before breaking the master branch.

[stoecker]

An automated patch testing would be possible with Trac as well and probably not even complicated.

I never saw the need for this. Failures often result from bitrotting and not from laziness. And errors from lazyness often come with structural errors, so failure to apply is a good indicator for further problems.

[stoecker]

Replying to GerdP:

BTW: Jenkins seems to have a problem.

While this is true in general I assume you mean something specific?

The i18n job at least worked a short time after it was fixed, but now it's broken with another problem...

[GerdP]

Replying to stoecker:

Replying to GerdP:

BTW: Jenkins seems to have a problem.

While this is true in general I assume you mean something specific?

The i18n job at least worked a short time after it was fixed, but now it's broken with another problem...

All jobs required much more time then usual and https://josm.openstreetmap.de/jenkins/job/JOSM/7022/ reported a lot of unit test failures with org.junit.runners.model.InitializationError
Short after I wrote this all jobs completed, so I guessed someone changed something.

[simon04]

Replying to stoecker:

• ask every plugin author to switch to the official JOSM GitLab.

Forget it :-)

Currently, there are 25 plugins at ​https://github.com/josm/?q=plugin

Since it's unlikely that all plugins can be migrated to GitLab, we'll have to deal with GitHub anyway. Here comes a probably unpopular suggestion: Let's migrate the remaining SVN plugins also to GitHub. Then most plugins are located at GitHub. Since we only host the repository (and possible pull requests) there, migrating the Git repository to another server is trivial.

[Stereo]

For what it's worth, and before someone complains about Microsoft: when investigating a GitHub actions problem at ​https://github.com/openstreetmap/operations/issues/381, I've found that GitHub is very open source friendly, that a lot of their code is actually free software, and that the team was nice and happy to help us and support us. I wouldn't have built the macOS releases workflow there otherwise.

[stoecker]

Since it's unlikely that all plugins can be migrated to GitLab, we'll have to deal with GitHub anyway.

No we have not. If GitLab is the future way for JOSM plugins then any plugin not hosted on JOSM gitlab will loose the status of JOSM maintained plugin and the author has to rely on himself for fixes, i18n and other updates.

Here comes a probably unpopular suggestion: Let's migrate the remaining SVN plugins also to GitHub. Then most plugins are located at GitHub.

That would finalize that JOSM will NOT be migrated to git.

Since we only host the repository (and possible pull requests) there, migrating the Git repository to another server is trivial.

No it's not. With any migration you will loose the complete history beside the code.

---

For what it's worth, and before someone complains about Microsoft: when investigating a GitHub actions problem at ​​https://github.com/openstreetmap/operations/issues/381, I've found that GitHub is very open source friendly, that a lot of their code is actually free software, and that the team was nice and happy to help us and support us. I wouldn't have built the macOS releases workflow there otherwise.

Microsoft and the relationship to OpenSource is a strange thing. First they fought it like it is the devil. Then they started to accept it. Lately they more and more support OpenSource. Whether that's a good thing or not only the future will show. I remain sceptical as I don't see yet how they will get enough money with OpenSource without tricks and vendor lock-in. Also you should never mix "a company" and "the people in a company". But the decision not to rely on external services has nothing to do with the fact that GitHub is now Microsoft owned.

[michael2402]

Here comes a probably unpopular suggestion: Let's migrate the remaining SVN plugins also to GitHub. Then most plugins are located at GitHub.

I am a fan of this. And especially streamlining the way plugins work there.
And by a look at https://josm.openstreetmap.de/wiki/PluginsSource there should not be that many remaining.

For example, pt_assistant is currently on gitlab+github (synced) and releases to github using github actions, runs the tests using gitlab -ci, does transfix using gitlabl-ci and some other stuff.

If we decide do migrate all plugins to github, we should make one repo for each plugin. We should then migrate all plugins to the same build system (Ant does not work for separate repos/separate ci runners, since it requires the main ant script).

Then we define the ​github workflow templates for the projects, like transfix and add them for each plugin as required. If we use github for releases and don't put them on the JOSM server, we would not need to change anything per plugin. If we want them on the JOSM server, we need to set a deployment access token for each plugin repository. So administrative work per repository is pretty minimal.

For anyone worried about migrations: Github actions are basically shell scripts that run in docker containers (they can also be VMs, but I would advise against that). So basically any other docker based build system can be easily migrated to, e.g. gitlab-ci or jenkins+kubernetes.

Replying to stoecker:

That would finalize that JOSM will NOT be migrated to git.

Why?

Since we only host the repository (and possible pull requests) there, migrating the Git repository to another server is trivial.

No it's not. With any migration you will loose the complete history beside the code.

There should be no relvant history beside the code.

[GerdP]

There should be no relvant history beside the code.

I always wonder if I have to update more than the plugin directory before running ant clean dist. My understanding is that build.xml depends on other files like build-common.xml and maybe also files in 00_* directories.
wiki:DevelopersGuide/DevelopingPlugins#Publishingthenewplugin is not clear about this.

[michael2402]

Replying to GerdP:

There should be no relvant history beside the code.

I always wonder if I have to update more than the plugin directory before running ant clean dist. My understanding is that build.xml depends on other files like build-common.xml and maybe also files in 00_* directories.
wiki:DevelopersGuide/DevelopingPlugins#Publishingthenewplugin is not clear about this.

Yes, this is a problem with the current way the plugins are set up.

They assume that you place JOSM at a given directory. You should always have checked out the JOSM min required version when compiling your plugin for a release. You should then check out at least JOSM latest and test if it would also compile against latest JOSM version.

As for the plugin common build script, I think you should have it at the latest version. Or the version, that works with your plugin (documented nowhere, exept if your plugin is in the ​plugin repo.

I can't document it better, since I am not using this workflow any more and any plugin I work with currently is already migrated to gradle, where the scripts do all that for you.

[stoecker]

Why?

It's no wonder we're having the same discussion again and again. You don't even try to understand my point of view.

[Don-vip]

I'm late to the party, but concerning the paid staff I agree with Dirk, and am completely opposed to talk about paid people for the infrastructure stuff or core development, for the reasons Dirk explained.

However I'm OK to see OSMF pay for tasks we (JOSM team) would like to see happening and clearly have no will/time/skills to do, such as:

• merge similar plugins together. For example: Mapillary and StreetSide, all the image plugins, etc.
• JOSM translations in major languages where we can't find anyone. The ​blog post I wrote two years ago is still relevant for Hindi, Bengali, Punjabi, Javanese, Wu, Telugu, Marathi, Tamil. Only Arabic translation did happen by itself.
• any other thing we all find boring / are not able to do ourselves

[michael2402]

If we don't want paid work in JOSM core, we should remove if from the Google summer of code wiki page:

​https://wiki.openstreetmap.org/wiki/Google_Summer_of_Code/2020/Project_ideas

[stoecker]

Replying to Don-vip:

However I'm OK to see OSMF pay for tasks we (JOSM team) would like to see happening and clearly have no will/time/skills to do, such as:

• merge similar plugins together. For example: Mapillary and StreetSide, all the image plugins, etc.
• JOSM translations in major languages where we can't find anyone. The ​blog post I wrote two years ago is still relevant for Hindi, Bengali, Punjabi, Javanese, Wu, Telugu, Marathi, Tamil. Only Arabic translation did happen by itself.
• any other thing we all find boring / are not able to do ourselves

It depends on the way how it is handled. I'm also not opposed to something like bounties, where someone get's outside money for a certain task to be done (e.g. Google of Code). But keeping the balance is not so easy, so I would keep it at a low level. JOSM is an extremely lifely project and even if there are some things we would like to have it is no major issue when they stay undone.

[Don-vip]

Agreed, I'm also OK with GSoC. Clearly my first preference for a paid job would be an Hindi/Bengali translation.

[taylor.smock]

I'm a bit late to this ticket, but:
Replying to Don-vip:

My cents:

• On the other hand, let's face it: ant itself is not properly maintained, as very few projects still use it. It's a shame to see major regressions in JUnit reports ​not being addressed. For this reason I think it would be good to have the possibility to additionally build JOSM using an up-to-date tool such as Maven (I don't like Gradle). This way we could restore the reporting mechanisms I setup in Jenkins long ago, that worked fine with JUnit 4+Ant junit task, and don't work anymore with JUnit 5 + Ant junitlauncher task, so that I had to commit an ugly workaround. But this is low priority.

I feel kind of bad about breaking some of the workflows that depended upon the JUnit4 output. :(

• I agree with everyone except Dirk on the need to switch to git over SVN. git is way faster and allow everyone to maintain their own branches without having to deal with patches. The patch approach is error-prone, difficult to read, and very time consuming (I often need to ask for new patch files that don't apply nicely, it seems tools behavirour vary between IDEs, OS, and so on). Like a lot of you, I use git daily at work and for other open source projects and SVN for JOSM is more and more a pain to use. This is one aspect of JOSM development that is not fun at all. This is not the main one, but still one of the reasons I lost interest in JOSM development this year.

Something else to note is that Mac has made it a bit harder to install subversion in recent releases. This also affects git svn, unfortunately. I was able to work around it, but a drive-by contributor might not (and my first patch might not have been submitted if I had had to take the time to do that -- I pretty much coded it during my "breaks" at work).

The target I'd like to see:

• ask every plugin author to switch to the official JOSM GitLab.

I actually started the MapWithAI plugin on GitLab with the intent of moving it to the JOSM GitLab, when it was set up. And it wasn't GitHub. So I'll be happy to move my plugins there. I don't know about anyone else though.

Replying to GerdP:

Debugging plugins rarely allows "hot code replacement" or whatever it is called, so a change in the plugin source typically means to stop JOSM; do ant clean dist for the plugin, manual copy of <plugin>.jar, start JOSM

I haven't had this issue except:

• When I change method signatures
• When I change object variables
• When I change a lambda

For the launch configuration, I have the appropriate plugin project selected (so JOSM-Mapillary for example), and then the main class is org.openstreetmap.josm.gui.MainApplication. I don't know if that is how you have done the launches. The only caveat is that you may run into missing resources (i.e., you added a new image), at which point you do need to run ant install or the equivalent.

Replying to Don-vip:

merge similar plugins together. For example: Mapillary and StreetSide, all the image plugins, etc.

On my personal TODO list (i.e., weekend project) is merging Mapillary and StreetSide. I'll probably have to post a patch for core to let people know that Mapillary/Streetside have been superseded. I don't think this will be too difficult, but...

[Don-vip]

Replying to taylor.smock:

I feel kind of bad about breaking some of the workflows that depended upon the JUnit4 output. :(

Don't be, you made huge progress :) This issue is almost gone now. After I ranted on Ant bugtracker, the maintainer committed a fix :) I'm starting to fix imagery entries again.

On my personal TODO list (i.e., weekend project) is merging Mapillary and StreetSide. I'll probably have to post a patch for core to let people know that Mapillary/Streetside have been superseded. I don't think this will be too difficult, but...

Thanks a lot!

[Stereo]

FreeBSD explains why they're moving to git: ​https://github.com/bsdimp/freebsd-git-docs/blob/main/git-why.md

[zelonewolf@…]

I would like to volunteer to work the migration from ant to gradle if the community desires it. I've had to make this transition in my professional life (as well as an svn->git transition), so I do have done this before, though I certainly wouldn't consider myself an expert.

The key benefit I see in gradle (really, maven-based build systems) is the automation of dependency management. In a gradle project, external jars are not checked into the repository; rather, the build script lists only the immediate dependencies. Any transitive dependencies (jars that depend on other jars) are downloaded at build/compile time. The only time you need to check in jars is when the jar doesn't exist in the maven ecosystem (which is quite rare these days -- usually only for a very bespoke library).

This means that when upgrading a dependency in gradle, all you have to do is change the dependency's version in the gradle build script, and voila! You're now building against the upgraded dependency *and all of its transitive dependencies are upgraded as well*. It will also alert you at compile time when there is a dependency collision (dependency A depends on C version x and dependency B depends on C version y).

Since jars aren't committed in a gradle build, this makes the size of the repo far smaller, as you don't have all the binary jars (and history of past versions of those jars) in the repository. If we are willing to cut the cord on change when making the svn->git migration, you could have a FAR smaller repository going forward.

So, I am willing to help with this, but if the community/maintainers here doesn't think it is useful to make this switch then obviously I won't bother. Having done this before, in my experience it makes more sense to switch to gradle first, and then switch to git with a clean repository that doesn't have jars in the repo's history.

[michael2402]

Replying to zelonewolf@…:

I would like to volunteer to work the migration from ant to gradle if the community desires it. I've had to make this transition in my professional life (as well as an svn->git transition), so I do have done this before, though I certainly wouldn't consider myself an expert.

I already did some professional projects, too. I even did JOSM, the gradle files are laying around attached as patch to some maven ticket I don't even bother to search now, since the decision back then was to still keep the two old build systems (ant + eclipse), so gradle would just add an additional one to maintain, would not save any work and would confuse people.

As for plugins: Many actively maintained plugins are already using gradle, or a hybrid gradpe+ant setup.

[Stereo]

SDL is also moving to git. They sound like they're pretty much in the same position as JOSM as regards to self-hosting.

​https://discourse.libsdl.org/t/sdl-moving-to-github/28700/5

The Register has some sarcasm:

​https://www.theregister.com/2021/02/10/sdl_moves_to_github/

[stoecker]

Mercurial. Aaaaah. I consider everything beside svn (centralised repo) and git (distributed repo, DVCS) dead ends today. All the other tools lost already. It maybe that svn will also lose against git and only DVCS survives, but that question is still open.

With the way he describes his hosting I'd also move to GitHub. Luckily our setup is extremely stable (well maybe except Jenkins which is sometimes awful) and server updates have never been problems.

[taylor.smock]

Replying to stoecker:

• [...] A bit complicated is only that Jenkins stuff mainly because of the missing separation between working data and configuration. And it seems to break easily when installing updates. I really would appreciate to have a store config and resetup jenkins script.

Jenkins will allow us to set up pipelines in SCM. See ​https://www.jenkins.io/doc/book/pipeline/getting-started/#defining-a-pipeline-in-scm .