Context Navigation

Modify ↓

#16944 closed enhancement (worksforme)

svn -> git migration script

Reported by:	wiktorn	Owned by:	team
Priority:	normal	Milestone:
Component:	unspecified	Version:
Keywords:	hack-weekend-2018-10 svn git migration script	Cc:

Description (last modified by wiktorn)

Split to 4 scripts to make it easier to restart script on errors

git_migrate-1-init.sh

#!/bin/sh

set -e

git init josm-tests
cd josm-tests
# create commit, to keep master branch clear
git commit -m 'init' --allow-empty

###
# define remotes
###

git svn init --trunk=. --prefix=svn_jmapviewer -R jmapviewer https://svn.openstreetmap.org/applications/viewer/jmapviewer
git svn init --trunk=. --prefix=svn_josm -R josm https://josm.openstreetmap.de/svn/trunk
# two separate remotes for commons as this is 2x ~10 minutes instead of 2+ houres to fetch
# git svn init --trunk=. --include-paths='commons/proper/jcs' --prefix=svn_commons_jcs/ -R svn_commons_jcs https://svn.apache.org/repos/asf/commons/proper/jcs/trunk/commons-jcs-core/
git svn init --trunk=. --prefix=svn_commons_jcs/ -R svn_commons_jcs https://svn.apache.org/repos/asf/commons/proper/jcs/trunk/commons-jcs-core/
# git svn init --trunk=. --prefix=svn_commons/ -R svn_commons https://svn.apache.org/repos/asf/
git svn init --trunk=trunk -b branches -t tags --prefix=svn_commons_logging/ -R svn_commons_logging https://svn.apache.org/repos/asf/commons/proper/logging/
git svn init --trunk=trunk -b branches -t tags --prefix=svn_commons_codec/ -R svn_commons_codec https://svn.apache.org/repos/asf/commons/proper/codec/
git svn init --trunk=trunk -b branches -t tags --prefix=svn_commons_bzip2/ -R svn_commons_bzip2 http://svn.apache.org/repos/asf/ant/core/

git svn init --trunk=. --prefix=svn_commons_bzip2_small/ -R svn_commons_bzip2_small http://svn.apache.org/repos/asf/ant/core/trunk/src/main/org/apache/tools/bzip2

git remote add commons_compress https://github.com/apache/commons-compress.git

echo Done

git_migrate-2-fetch.sh

#!/bin/sh

set -e

cd josm-tests
###
# fetch from remotes
###
git svn fetch jmapviewer
git svn fetch josm
# git svn fetch --log-window-size=100000 svn_commons
git svn fetch --log-window-size=100000 svn_commons_jcs
git svn fetch --log-window-size=100000 svn_commons_logging
git svn fetch --log-window-size=100000 svn_commons_codec
# git svn fetch --log-window-size=100000 svn_commons_bzip2
git svn fetch --log-window-size=100000 svn_commons_bzip2_small
git fetch commons_compress

echo Done

git_migrate-3-prepare.sh

#!/bin/sh

set -e

cd josm-tests
###
# Reorganize externals
###
# TODO - for all - squash all commits before introduction to JOSM to one
# remove files, on removal in JOSM

# JMapViewer
git branch -f svn_jmapviewer_rewrite remotes/svn_jmapviewertrunk
git filter-branch -f --prune-empty --tree-filter '
  if [ -d src/org/openstreetmap/gui/jmapviewer ] ; then
      git mv src src.new &&
      find . -depth 1 |
      grep -v ^./src.new | grep -v ^./.git$ |
      xargs -r git rm -fr &&
      mkdir -p src/org/openstreetmap/gui &&
      git add src/org/openstreetmap/gui &&
      git mv src.new/org/openstreetmap/gui/jmapviewer src/org/openstreetmap/gui &&
      git rm -rf --ignore-unmatch src.new
  else
      find . -depth 1 |
      grep -v ^./.git$ |
      xargs -r git rm -fr
  fi' \
  svn_jmapviewer_rewrite

# Apache Commons Compress
# Linearlize commons_compress history
git branch -f svn_commons_compress_rewrite remotes/commons_compress/master
# linearlize history
git filter-branch -f --parent-filter 'cut -f 2,3 -d " "' svn_commons_compress_rewrite
# Extract externals
git filter-branch -f --prune-empty --tree-filter '
  if [ -d src/main/java ] ; then
      git mv src src.new &&
      find . -depth 1 |
      grep -v ^./src.new | grep -v ^./.git$ |
      xargs -r git rm -fr &&
      git mv src.new/main/java src &&
      git rm -rf --ignore-unmatch src.new
  else
      find . -depth 1 |
      grep -v ^./.git$ |
      xargs -r git rm -fr
  fi' \
  svn_commons_compress_rewrite


# Apache Commons JCS
git branch -f svn_commons_jcstrunk_rewrite remotes/svn_commons_jcs/trunk
# git filter-branch -f --prune-empty --tree-filter '
#   if [ -d trunk/commons-jcs-core/src/main/java/ ] ; then
#       git rm -rf src ;
#       git mv trunk/commons-jcs-core/src/main/java src &&
#       find . -depth 1 |
#       grep -v ^./src$ | grep -v ^./.git$ |
#       xargs -r git rm -fr
#   else
#       find . -depth 1 |
#       grep -v ^./.git$ |
#       xargs -r git rm -fr
#   fi' \
#   svn_commons_jcstrunk_rewrite
git filter-branch -f --prune-empty --tree-filter '
  if [ -d src/main/java/ ] ; then
      git mv src src.new &&
      find . -depth 1 |
      grep -v ^./src.new | grep -v ^./.git$ |
      xargs -r git rm -fr &&
      git mv src.new/main/java src &&
      git rm -rf --ignore-unmatch src.new
  else
      find . -depth 1 |
      grep -v ^./.git$ |
      xargs -r git rm -fr
  fi' \
  svn_commons_jcstrunk_rewrite

# Apache Commons Logging
git branch -f svn_commons_loggingtrunk_rewrite remotes/svn_commons_logging/trunk
git filter-branch -f --prune-empty --tree-filter '
  if [ -d src/main/java ] ; then
      git mv src src.new &&
      find . -depth 1 |
      grep -v ^./src.new | grep -v ^./.git$ |
      xargs -r git rm -fr &&
      git mv src.new/main/java src &&
      git rm -rf --ignore-unmatch src.new
  else
      find . -depth 1 |
      grep -v ^./.git$ |
      xargs -r git rm -fr
  fi' \
  svn_commons_loggingtrunk_rewrite

# TODO: Apache Codecs
# svn_commons_codec
git branch -f svn_commons_codec_rewrite remotes/svn_commons_codec/trunk
git filter-branch -f --prune-empty --tree-filter '
  if [ -d src/main/java ] ; then
      git mv src src.new &&
      find . -depth 1 |
      grep -v ^./src.new | grep -v ^./.git$ |
      xargs -r git rm -fr &&
      git mv src.new/main/java src &&
      git rm -rf --ignore-unmatch src.new
  elif [ -d src/java ] ; then
      git mv src src.new &&
      find . -depth 1 |
      grep -v ^./src.new | grep -v ^./.git$ |
      xargs -r git rm -fr &&
      git mv src.new/java src &&
      git rm -rf --ignore-unmatch src.new
  else
      find . -depth 1 |
      grep -v ^./.git$ |
      xargs -r git rm -fr
  fi' \
  svn_commons_codec_rewrite

git branch -f svn_commons_bzip2_rewrite remotes/svn_commons_bzip2_small/trunk
git filter-branch -f --prune-empty --tree-filter '
  mkdir -p src/org/apache/tools/bzip2
  git mv *java src/org/apache/tools/bzip2
  ' \
  svn_commons_bzip2_rewrite

git_migrate-4-adapt-history.sh

#!/bin/sh

set -e

cd josm-tests

for i in svn_jmapviewer svn_commons_jcstrunk svn_commons_loggingtrunk svn_commons_codec svn_commons_compress svn_commons_bzip2 ; do
  git branch -f "${i}_clean" "${i}_rewrite"
  # git rebase $(git log --reverse --oneline --no-abbrev-commit "${i}_clean" | head -n 1 | cut -d ' ' -f 1) "${i}_clean" || exit 1
done

git branch -f svn_commons_compress_compressors_clean svn_commons_compress_rewrite
git branch -f svn_commons_compress_utils_clean svn_commons_compress_rewrite


function josm_ver_to_timestamp {
  git log --grep "git-svn-id: https://josm.openstreetmap.de/svn/trunk@${1}" --pretty="format:%at" svn_josmtrunk
}

function first_before {
  BRANCH="${1}"
  TIMESTAMP="${2}"
  COMMIT_NO=$(( $( git log --pretty="format:%at %H" "${BRANCH}" | sort -n |  awk -F ' ' "\$1 > ${TIMESTAMP} {print NR} {} " | head -n 1) - 1 ))
  git log --pretty="format:%at %H" ${BRANCH} | sort -n | awk -F ' ' "NR == ${COMMIT_NO} {print \$2}"
}

function first_after {
  BRANCH="${1}"
  TIMESTAMP="${2}"
  #COMMIT_NO=$( git log --pretty="format:%at %H" "${BRANCH}" | sort -n |  awk -F ' ' "\$1 < ${TIMESTAMP} {print \$2} {} " | head -n 1)
  git log --pretty="format:%at %H" "${BRANCH}" | sort -n |  awk -F ' ' "\$1 < ${TIMESTAMP} {print \$2} {} " | tail -n 1
  #git log --pretty="format:%at %H" ${BRANCH} | sort -n | awk -F ' ' "NR == ${COMMIT_NO} {print \$2}"
}

function extract_timestamp {
    echo $1 |  sed -E -e "s%[^0-9]*([0-9]*)[^0-9]*.*%\1%"
}

function truncate_from_to {
    BRANCH="$1"
    START_TS="$2"
    STOP_TS="$3"
    echo "Working on ${BRANCH}"
    if [ -z "${BRANCH}" ] ; then
         echo No branch provided
         exit 1;
    fi
    if [ -z "${START_TS}" ] ; then
         echo no STATRT_TS provided
         exit 1;
    fi

    COMMIT_ID=$(first_before "${BRANCH}" "${START_TS}")

    if [ -z "${COMMIT_ID}" ] ; then
        echo Unable to find commit on "${BRANCH}" before "${START_TS}"
        exit 1
    fi
    export BRANCH
    export START_TS
    export COMMIT_ID
    git filter-branch -f --commit-filter '
      if [ "${GIT_COMMIT}" == "${COMMIT_ID}" ] ; then
        GIT_AUTHOR_DATE="${START_TS}" git commit-tree -m "Initial import of ${BRANCH}" "$@"
      elif [ $(echo "${GIT_AUTHOR_DATE}" | sed -E -e "s%[^0-9]*([0-9]*)[^0-9]*.*%\1%" )  -lt "${START_TS}" ] ; then
        skip_commit "$@"
      else
        git commit-tree "$@"
      fi
    ' ${BRANCH}


    if [ -n "${STOP_TS}" ] ; then
        COMMIT_ID=$(first_after "${BRANCH}" "${STOP_TS}")
        export COMMIT_ID
        export STOP_TS
        git filter-branch -f --tree-filter '
          if [ "${GIT_COMMIT}" == "${COMMIT_ID}" ]; then
             find . -depth 1 | grep -v ^./git$ | xargs -r git rm -rf
          fi
          ' --commit-filter '
          if [ "${GIT_COMMIT}" == "${COMMIT_ID}" ] ; then
            GIT_AUTHOR_DATE="${STOP_TS}" git commit-tree -m "Remove ${BRANCH}" "$@"
          elif [ $(echo "${GIT_AUTHOR_DATE}" | sed -E -e "s%[^0-9]*([0-9]*)[^0-9]*.*%\1%" ) -gt "${STOP_TS}" ] ; then
              skip_commit "$@"
          else
              git commit-tree "$@"
          fi
        ' ${BRANCH}
    fi
}

###
# Clean compress_compressors
###
git filter-branch -f --tree-filter '
  find src/org/apache/commons/compress -depth 1 | grep -v compressors | xargs -r git rm -rf
' svn_commons_compress_compressors_clean

git filter-branch -f --tree-filter '
  find src/org/apache/commons/compress -depth 1 | grep -v utils | xargs -r git rm -rf
' svn_commons_compress_utils_clean

truncate_from_to svn_commons_codec_clean $(josm_ver_to_timestamp 2796) $(josm_ver_to_timestamp 8149)
truncate_from_to svn_commons_bzip2_clean $(josm_ver_to_timestamp 2796) $(josm_ver_to_timestamp 7867)
truncate_from_to svn_commons_compress_compressors_clean $(josm_ver_to_timestamp 7867) $(josm_ver_to_timestamp 12499)
truncate_from_to svn_commons_compress_utils_clean $(josm_ver_to_timestamp 11569) $(josm_ver_to_timestamp 12499)
truncate_from_to svn_commons_compress_clean $(josm_ver_to_timestamp 12500)
truncate_from_to svn_commons_jcstrunk_clean $(josm_ver_to_timestamp 8168)
truncate_from_to svn_commons_loggingtrunk_clean $(josm_ver_to_timestamp 8168)

# get history of svn externals:
# svn propget -R svn:externals src/org/apache
# then e.g.
# svn log src/org/apache/commons
# svn propget -r 7867 -R svn:externals src/org/apache
#  since 2796 codec - http://svn.apache.org/repos/asf/commons/proper/codec/trunk/src/java/org/apache/commons/codec
#             tools/bzip2       - http://svn.apache.org/repos/asf/ant/core/trunk/src/main/org/apache/tools/bzip2
#  since 4460 - codec - http://svn.apache.org/repos/asf/commons/proper/codec/trunk/src/main/java/org/apache/commons/codec
#  since 7867 - tools/bzip2 -> compress ->  http://svn.apache.org/repos/asf/commons/proper/compress/trunk/src/main/java/org/apache/commons/compress/compressors
# since 8149 - no codec
# since 8168 - jcs - http://svn.apache.org/repos/asf/commons/proper/jcs/trunk/commons-jcs-core/src/main/java/org/apache/commons/jcs
#            - logging - http://svn.apache.org/repos/asf/commons/proper/logging/trunk/src/main/java/org/apache/commons/logging logging
# since 8698 - compress ->  https://github.com/apache/commons-compress/trunk/src/main/java/org/apache/commons/compress/compressors
# since 11569 - utils -> https://github.com/apache/commons-compress/trunk/src/main/java/org/apache/commons/compress/utils
# since 12500 - compress -> https://github.com/apache/commons-compress/trunk/src/main/java/org/apache/commons/compress compress

git_migrate-5-merge.sh

#!/bin/sh

set -e

cd josm-tests

###
# Merge JOSM repository with externals, commits ordered by date
###

# each git log needs echo at the end because first/last commit will be lost
(
  git log --pretty="format:%at %H %s" remotes/svn_josmtrunk ; echo
  git log --pretty="format:%at %H %s" svn_jmapviewer_clean ; echo
  git log --pretty="format:%at %H %s" svn_commons_jcstrunk_clean ; echo
  git log --pretty="format:%at %H %s" svn_commons_loggingtrunk_clean ; echo
  git log --pretty="format:%at %H %s" svn_commons_codec_clean ; echo
  git log --first-parent --pretty="format:%at %H %s" svn_commons_compress_clean ; echo
  git log --first-parent --pretty="format:%at %H %s" svn_commons_compress_compressors_clean; echo
  git log --first-parent --pretty="format:%at %H %s" svn_commons_compress_utils_clean ; echo
  git log --pretty="format:%at %H %s" svn_commons_bzip2_clean; echo
) | sort -n | cut -d ' ' -f2 | sed -e 's/^/p /' > ../rebase_config

git config merge.renamelimit 5000
git checkout master
git rebase --keep-empty -i $(git log --reverse --oneline --no-abbrev-commit | head -n 1 | cut -d ' ' -f 1)

# helpful one-liner: git diff --name-only --diff-filter=U | xargs git checkout  ...

# remove empty commits
git rebase $(git log --reverse --oneline --no-abbrev-commit | head -n 1 | cut -d ' ' -f 1)

###
# Move jar files to Git LFS
###
git lfs migrate import --include="*.jar" --include-ref=refs/heads/master
git reflog expire --expire-unreachable=now --all
git gc --prune=now

Attachments (0)

Change History (13)

comment:1 Changed 5 years ago by wiktorn

This script merges all (?) our externals into one git, ordering commits by date. Need some work on Apache Commons Compress, as date ordering is not always what should be.

Result of this script is downloadable from: http://gitlab.vink.pl/w/josm-tests.git
Repository has size of: 157MB
LFS objects: 918MB
Full clone (including current version of jar files): 233MB

comment:2 Changed 5 years ago by wiktorn

Description:	modified (diff)

Split to 4 scripts. Apart from 1, all may be restarted.

I'll also try to check if JOSM actually builds from this (like every 100 commit or so).

comment:3 Changed 5 years ago by wiktorn

Description:	modified (diff)

comment:4 Changed 5 years ago by wiktorn

Description:	modified (diff)

comment:5 Changed 5 years ago by wiktorn

I've run following script to check how does the old versions build:

#!/bin/bash

for i in $(cd josm-tests-builds && git log --oneline --reverse --no-abbrev | awk '0 == NR % 100' | cut -d ' ' -f  1) ; do
        (
                cd josm-tests-builds &&
                git reset --hard && git clean -f
                git checkout $i &&
                ant clean compile clean > ../report/${i}.err && mv ../report/${i}.err ../report/${i}.ok
        )
done

This gets every 100th commit. As GIT history starts before JOSM (I haven't squashed those commits, not sure if it's necessary), few first builds fail, as there is no build.xml.

There are also different failures. Probably due to changes between version that was used and JDK8:

error: unreported exception IOException; must be caught or declared to be thrown
error: name clash: put(A,B) in MultiMap and put(K,V) in HashMap have the same erasure, yet neither overrides the other
error: cannot access JPEGDecodeParam

Missing externals:

error: package org.apache.commons.codec.binary does not exist
error: package org.apache.commons.compress.compressors.bzip2 does not exist
error: package org.apache.commons.compress.compressors.xz does not exist

Problems in earlier versions of JCS:

error: method does not override or implement a method from a supertype
error: package org.apache.velocity does not exist
error: package javax.servlet.http does not exist

Problems in logging:

error: package org.apache.avalon.framework.logger does not exist
error: package org.apache.log4j does not exist
error: package org.apache.log does not exist

So the lesson here is, that externals should be squashed to the time they were introduced, and if they were removed - remove them also from our repo.

Regardless of the above, 20 builds succeeded :-)

comment:6 Changed 5 years ago by Don-vip

When we reached commit 10.000 I re-run every past build to recreate SonarQube history.

I had two scripts:

<project name="josm" default="test-it" basedir="." xmlns:jacoco="antlib:org.jacoco.ant">

  <path id="test.classpath">
    <pathelement path="build"/>
    <pathelement path="test/build"/>
    <fileset dir="dist" erroronmissingdir="false">
      <include name="josm-custom.jar"/>
    </fileset>
    <fileset dir="lib" erroronmissingdir="false">
      <include name="**/*.jar"/>
    </fileset>
    <fileset dir="test/lib" erroronmissingdir="false">
      <include name="**/*.jar"/>
      <exclude name="junit/**/*.jar"/>
      <exclude name="fest/**/*.jar"/>
    </fileset>
    <fileset dir="tools" erroronmissingdir="false">
      <include name="findbugs/annotations.jar"/>
      <include name="groovy-all*.jar"/>
    </fileset>
    <fileset dir="/home/josm/auto/svn_josm/core/test/lib">
      <include name="junit/**/*.jar"/>
      <include name="fest/**/*.jar"/>
    </fileset>
  </path>

  <target name="test">
    <taskdef uri="antlib:org.jacoco.ant" resource="org/jacoco/ant/antlib.xml" classpath="/home/josm/auto/svn_josm/core/tools/jacocoant.jar" />
    <jacoco:coverage destfile="test/jacoco.exec">
      <junit printsummary="yes" fork="true" forkmode="once">
        <jvmarg value="-Dfile.encoding=UTF-8"/>
        <sysproperty key="josm.home" value="test/config/unit-josm.home"/>
        <sysproperty key="josm.test.data" value="test/data"/>
        <sysproperty key="java.awt.headless" value="true"/>
        <classpath>
          <path refid="test.classpath"/>
          <pathelement path="test/config"/>
        </classpath>
        <formatter type="plain"/>
        <formatter type="xml"/>
        <batchtest fork="yes" todir="test/report">
          <fileset dir="test" includes="org/**/*Test.java" erroronmissingdir="false"/>
          <fileset dir="test/unit" includes="org/**/*Test.java" erroronmissingdir="false"/>
          <fileset dir="test/functional" includes="org/**/*Test.java" erroronmissingdir="false"/>
          <fileset dir="test/performance" includes="org/**/*Test.java" erroronmissingdir="false"/>
        </batchtest>
      </junit>
    </jacoco:coverage>
  </target>

  <target name="test-it" depends="test">
    <jacoco:coverage destfile="test/jacocoIT.exec">
      <junit printsummary="yes" fork="true" forkmode="once">
        <jvmarg value="-Dfile.encoding=UTF-8"/>
        <sysproperty key="josm.home" value="test/config/unit-josm.home"/>
        <sysproperty key="josm.test.data" value="test/data"/>
        <sysproperty key="java.awt.headless" value="true"/>
        <classpath>
          <path refid="test.classpath"/>
          <pathelement path="test/config"/>
        </classpath>
        <formatter type="plain"/>
        <formatter type="xml"/>
        <batchtest fork="yes" todir="test/report">
          <fileset dir="test" includes="org/**/*TestIT.java" erroronmissingdir="false"/>
          <fileset dir="test/unit" includes="org/**/*TestIT.java" erroronmissingdir="false"/>
          <fileset dir="test/functional" includes="org/**/*TestIT.java" erroronmissingdir="false"/>
          <fileset dir="test/performance" includes="org/**/*TestIT.java" erroronmissingdir="false"/>
        </batchtest>
      </junit>
    </jacoco:coverage>
  </target>
</project>

=> This custom build.xml used up-to-date tools stored outside the SVN repository

#!/bin/bash

# ------------------------------------------------------------------------- #
# JOSM Historical data feeder for SonarQube                                 #
# ------------------------------------------------------------------------- #
# Feeds historical data of JOSM development to a SonarQube instance         #
#                                                                           #
# This script takes care of the following events in JOSM history            #
# https://josm.openstreetmap.de/changeset/109/josm : add build.xml          #
# https://josm.openstreetmap.de/changeset/111/josm : remove debug in build  #
# https://josm.openstreetmap.de/changeset/269/josm : use JUnit 4            #
# https://josm.openstreetmap.de/changeset/321/josm : trunk directory        #
# https://josm.openstreetmap.de/changeset/413/josm : add debug in build     #
# https://josm.openstreetmap.de/changeset/1156/josm: remove unit tests      #
# https://josm.openstreetmap.de/changeset/1622/josm: add unit tests         #
# https://josm.openstreetmap.de/changeset/1751/josm: add test target        #
# https://josm.openstreetmap.de/changeset/1752/josm: add wrong env file     #
# https://josm.openstreetmap.de/changeset/3094/josm: wrong Java 6 commit    #
# https://josm.openstreetmap.de/changeset/3105/josm: revert Java 6 commit   #
# https://josm.openstreetmap.de/changeset/3321/josm: wrong Java 6 commit    #
# https://josm.openstreetmap.de/changeset/3336/josm: revert Java 6 commit   #
# https://josm.openstreetmap.de/changeset/3378/josm: switch to Java 6       #
# https://josm.openstreetmap.de/changeset/7001/josm: switch to Java 7       #
# https://josm.openstreetmap.de/changeset/9501/josm: unit/IT tests          #
# ------------------------------------------------------------------------- #

OSM_USERNAME="xxx"
OSM_PASSWORD="xxx"

WORKDIR="/home/josm/team/Vincent/historic"
RUNNER="/opt/sonar-scanner-2.5/bin/sonar-runner"
JDK5_HOME="/opt/jdk1.5.0_22"
JDK6_HOME="/opt/jdk1.6.0_45"
JDK7_HOME="/opt/jdk1.7.0_80"
JDK8_HOME="/usr/lib/jvm/java-8-openjdk-amd64"
JDK9_HOME="/usr/lib/jvm/java-9-openjdk-amd64"

JUNIT3="/usr/share/java/junit-3.8.2.jar"
JUNIT4="/home/josm/auto/svn_josm/core/test/lib/junit/junit-4.12.jar:/home/josm/auto/svn_josm/core/test/lib/junit/hamcrest-core-1.3.jar"

JACOCO="/home/josm/auto/svn_josm/core/tools/jacocoant.jar"

HOST_URL="https://josm.openstreetmap.de/sonar"
PROJECT_KEY="josm_releases"
PROJECT_NAME="JOSM_Releases"

#HOST_URL="https://sonarqube.com"
#PROJECT_KEY="josm"
#PROJECT_NAME="JOSM"
#PROJECT_TOKEN="xxx"

TRAC_URL="https://josm.openstreetmap.de"
TRAC_USER="xxx"
TRAC_PASSWORD="xxx"

function manual_tests {
  rev=$1
  java=$2
  jdk=$3
  libs=$4

  rm -Rf "$WORKDIR/$rev/test/build" "$WORKDIR/$rev/test/report" "$WORKDIR/$rev/test/jacoco.exec"
  mkdir "$WORKDIR/$rev/test/build"
  if [ $rev -lt 269 ] ; then
    junit=$JUNIT3
    javac=$JDK5_HOME/bin/javac
  else
    junit=$JUNIT4
  fi
  if [ -d "$WORKDIR/$rev/test/unit" ] ; then
    testdir="$WORKDIR/$rev/test/unit"
    for d in "functional" "performance" ; do
      if [ -d "$WORKDIR/$rev/test/$d" ] ; then
        testdir="$testdir:$WORKDIR/$rev/test/$d"
      fi
    done
  else
    testdir="$WORKDIR/$rev/test"
  fi
  td="/home/josm/auto/svn_josm/core/test/lib"
  testlibs="$libs:$WORKDIR/$rev/lib/jfcunit.jar:$td/fest/MRJToolkitStubs-1.0.jar:$td/fest/debug-1.0.jar:$td/fest/fest-assert-1.0.jar:$td/fest/fest-reflect-1.1.jar:$td/fest/fest-swing-1.1.jar:$td/fest/fest-util-1.0.jar:$td/fest/jcip-annotations-1.0.jar:$td/unitils-core/unitils-core-3.4.2.jar:$td/unitils-core/ognl-2.6.9.jar:$td/unitils-core/commons-logging-1.1.jar:$td/unitils-core/commons-lang-2.3.jar:$td/unitils-core/commons-collections-3.2.jar:$WORKDIR/$rev/test/lib/equalsverifier-1.7.6.jar:$WORKDIR/$rev/test/lib/equalsverifier-2.0.2.jar:$WORKDIR/$rev/test/lib/equalsverifier-2.1.5.jar:$WORKDIR/$rev/tools/findbugs/annotations.jar"
  IFS=':' read -ra ADDR <<< "$testdir"
  for d in "${ADDR[@]}"; do
    for f in `find $d -name *Test.java` ; do
      $jdk/bin/javac -encoding UTF8 -source $java -target $java -d "$WORKDIR/$rev/test/build" -cp "$WORKDIR/$rev/build:$junit:$testdir:$testlibs" $f
      if [ ! $? -eq 0 ] && [ $rev -ge 1622 ] && [ $d != "$WORKDIR/$rev/test/functional" ] ; then
        echo "Unexpected compilation error"
        exit 100
      fi
    done
  done
  mkdir "$WORKDIR/$rev/test/report"
  if [ -f "$WORKDIR/$rev/test/config/preferences.template.xml" ] ; then
    for fam in "unit" "functional" "performance" ; do
      if [ -d "$WORKDIR/$rev/test/config/$fam-josm.home" ] ; then
        sed "s/@OSM_USERNAME@/$OSM_USERNAME/g; s/@OSM_PASSWORD@/$OSM_PASSWORD/g" $WORKDIR/$rev/test/config/preferences.template.xml > $WORKDIR/$rev/test/config/$fam-josm.home/preferences.xml
      fi
    done
  fi
  ant -Dbasedir="$WORKDIR/$rev" -f "$WORKDIR/build_test.xml" | grep --line-buffered FAILED
}

# Almost all tested releases (not the very close ones)
for rev in 14 25 35 40 43 53 61 100 166 285 802 869 898 929 970 1010 1065 1091 1097 1137 1178 1198 1212 1344 1387 1437 1460 1479 1504 1515 1529 1566 1607 1669 1788 1981 2221 2255 2552 2554 2561 3094 3208 3329 3376 3514 3592 3701 3751 3965 3966 4021 4064 4201 4223 4279 4399 4487 4550 4667 4878 5047 5181 5207 5210 5267 5315 5356 5485 5531 5576 5608 5697 5759 5836 5939 5990 6060 6115 6238 6388 6502 6766 6767 6891 6950 7000 7182 7287 7347 7480 7588 7643 7777 7906 7995 8109 8159 8279 8339 8491 8677 8800 8969 9060 9229 9329 9900 9979 10168 10301 10526 10786 10966
do
  baseurl="http://josm.openstreetmap.de/svn"
  if [ $rev -gt 321 ] ; then
    baseurl="$baseurl/trunk"
  fi

  log="$WORKDIR/$rev/log.txt"
  if [ -f $log ] ; then
    rm -f $log
  fi

  if [ ! -d $WORKDIR/$rev ] ; then
    svn co -q $baseurl@$rev $WORKDIR/$rev #|| exit 1
  fi
  cd $WORKDIR/$rev || exit 2
  date=`svn info | grep "Last Changed Date" | cut -d" " -f4`

  # replace svn:externals that do not exist anymore
  if [ -d "$WORKDIR/$rev/src/org/apache/commons" ] ; then
    codec=`svn propget svn:externals $WORKDIR/$rev/src/org/apache/commons | grep codec | cut -d" " -f2`
    if [ "$codec" = "http://svn.apache.org/repos/asf/commons/proper/codec/trunk/src/java/org/apache/commons/codec" ] ; then
      svn co -q http://svn.apache.org/repos/asf/commons/proper/codec/trunk/src/java/org/apache/commons/codec@1173286 $WORKDIR/$rev/src/org/apache/commons/codec || exit 2
    fi
    if [ -d "$WORKDIR/$rev/src/org/apache/commons/compress" ] ; then
      compressors=`svn propget svn:externals $WORKDIR/$rev/src/org/apache/commons/compress | grep compressors | cut -d" " -f2`
      if [ "$compressors" = "http://svn.apache.org/repos/asf/commons/proper/compress/trunk/src/main/java/org/apache/commons/compress/compressors" ] ; then
        svn co -q https://github.com/apache/commons-compress/trunk/src/main/java/org/apache/commons/compress/compressors $WORKDIR/$rev/src/org/apache/commons/compress/compressors || exit 2
      fi
    fi
  fi

  # checkout svn:externals at the correct revision
  for d in "data" "data_nodist" "eclipse" "images" "images_nodist" "lib" "linux" "macosx" "netbeans" "patches" "presets" "resources" "scripts" "src" "styles" "styles_nodist" "test" "tools" "windows" ; do
    if [ -d $WORKDIR/$rev/$d ] ; then
      find $WORKDIR/$rev/$d -name .svn -execdir svn update -q -r {$date} \;
    fi
  done

  # identify java version
  if [[ $rev -lt 3378 && ( $rev -lt 3094 || ( $rev -ge 3105 && ( $rev -lt 3321 || $rev -ge 3336 ) ) ) ]] ; then
    java="1.5"
    jdk=$JDK5_HOME
  elif [ $rev -lt 7001 ] ; then
    java="1.6"
    jdk=$JDK6_HOME
  elif [ $rev -lt 10580 ] ; then
    java="1.7"
    jdk=$JDK7_HOME
  else
    java="1.8"
    jdk=$JDK8_HOME
  fi

  # workaround for tests placed in source folders (r100)
  if [ -d "$WORKDIR/$rev/src/org/openstreetmap/josm/test" ] && [ ! -d "$WORKDIR/$rev/test" ] ; then
    mkdir -p "$WORKDIR/$rev/test/org/openstreetmap/josm"
    mv "$WORKDIR/$rev/src/org/openstreetmap/josm/test" "$WORKDIR/$rev/test/org/openstreetmap/josm/"
  fi

  # remove invalid unit test env file
  if [ -f "$WORKDIR/$rev/test/config/test-unit-env.properties" ] ; then
    rm -f "$WORKDIR/$rev/test/config/test-unit-env.properties"
  fi

  # compile and run tests
  export JAVACMD=$jdk/bin/java
  if [ $rev -ge 413 ] ; then # we have a build.xml with debug compilation
    if [ $rev -lt 9842 ] ; then # but we run tests manually to be sure to use jacoco
      cleantargets="clean"
      targets="dist"
    else
      cleantargets="clean test-clean"
      targets="dist test"
    fi
    ant -S $cleantargets $targets || exit 3
    # we have unit tests until r1156, working fine in earlier revisions, then slowly abandoned, then restored in r1622, but without target until r1751, and jacoco latest change in 9842
    if [[ $rev -lt 1156 || ( $rev -ge 1622 && $rev -lt 9842 ) ]] ; then
      manual_tests $rev $java $jdk $libs
    fi
  else # manual compilation, we don't have any build.xml, or a build without debug compilation
    libdir="$WORKDIR/$rev/lib"
    libs="$libdir/jdom.jar:$libdir/MinML2.jar:$libdir/metadata-extractor-2.3.1.jar:$libdir/gettext-commons-0.9.jar"
    rm -Rf "$WORKDIR/$rev/build"
    mkdir "$WORKDIR/$rev/build"
    $JDK5_HOME/bin/javac -encoding UTF8 -source $java -target $java -d "$WORKDIR/$rev/build" -cp "$WORKDIR/$rev/src:$libs" `find src/org/openstreetmap/josm -name Main*.java` || exit 4
    if [ -f "$WORKDIR/$rev/src/org/openstreetmap/josm/data/conflict/Merger.java" ] ; then
      $JDK5_HOME/bin/javac -encoding UTF8 -source $java -target $java -d "$WORKDIR/$rev/build" -cp "$WORKDIR/$rev/src:$libs" "$WORKDIR/$rev/src/org/openstreetmap/josm/data/conflict/Merger.java" || exit 5
    fi
    if [ -d "$WORKDIR/$rev/test" ] ; then # manual tests
      manual_tests $rev $java $jdk $libs
    fi
  fi
  unset JAVACMD

  # identify JOSM version
  if [ $rev -eq 14 ] ; then
    version="1.0b1"
  elif [ $rev -eq 25 ] ; then
    version="1.0b2"
  elif [ $rev -eq 35 ] ; then
    version="1.0rc1"
  elif [ $rev -eq 40 ] ; then
    version="1.0rc2"
  elif [ $rev -eq 43 ] ; then
    version="1.0"
  elif [ $rev -eq 53 ] ; then
    version="1.1"
  elif [ $rev -eq 61 ] ; then
    version="1.2"
  elif [ $rev -eq 100 ] ; then
    version="1.3"
  elif [ $rev -eq 166 ] ; then
    version="1.4"
  elif [ $rev -eq 285 ] ; then
    version="1.5"
  else
    version=$rev
  fi

  echo "Analyzing r$rev - $date - $version - java $java"

  # List of source folders
  sources="src"
  for d in "data" "images" "linux" "macosx" "presets" "resources" "scripts" "styles" "windows" ; do
    if [ -d $WORKDIR/$rev/$d ] ; then
      sources="$sources,$d"
    fi
  done

  # SonarQube analysis parameters
  runner_args="-Dsonar.host.url=$HOST_URL -Dsonar.projectKey=$PROJECT_KEY -Dsonar.projectName=$PROJECT_NAME -Dsonar.projectVersion=$version -Dsonar.projectDate=$date -Dsonar.sources=$sources -Dsonar.sourceEncoding=UTF-8 -Dsonar.java.source=$java -Dsonar.java.binaries=build -Dsonar.scm.url=scm:$baseurl -Dsonar.findbugs.timeout=1200000 -Dsonar.trac.url=$TRAC_URL -Dsonar.trac.username.secured=$TRAC_USER -Dsonar.trac.password.secured=$TRAC_PASSWORD -Dsonar.links.homepage=$TRAC_URL -Dsonar.links.ci=$TRAC_URL/jenkins -Dsonar.links.issue=$TRAC_URL/report -Dsonar.links.scm=$TRAC_URL/browser/josm/trunk -Dsonar.links.scm_dev=https://josm.openstreetmap.de/svn/trunk -Djava_version=${java/1./} -Dsonar.inclusions=src/org/openstreetmap/josm/**/*.java,**/*.groovy,**/*.xml,**/*.xsd,**/*.css,**/*.osm,**/*.mapcss,**/*.properties,**/*.json,**/*.js,**/*.cfg -Dsonar.exclusions=src/org/openstreetmap/josm/gui/mappaint/mapcss/parsergen/**/*.java,src/org/openstreetmap/josm/data/imagery/types/*.java,data/overpass-turbo-ffs.js,data/overpass-wizard.js,data/validator/opening_hours.js,macosx/JOSM.app/Contents/Info.plist_template.xml -Dsonar.test.inclusions=test/**/*.java,test/**/*.groovy"

# TODO does not work :'(
# -Djava_version=${java/1./}

  if [ -d "$WORKDIR/$rev/test" ] ; then
    tests=""
    for t in "unit" "functional" "performance" ; do
      if [ -d "$WORKDIR/$rev/test/$t" ] ; then
        if [ ! -z "$tests" ] ; then
          tests="$tests,"
        fi
        tests="${tests}test/$t"
      fi
    done
    if [ -z "$tests" ] ; then
      tests="test"
    fi
    runner_args="$runner_args -Dsonar.tests=$tests"

    if [ -d "$WORKDIR/$rev/test/lib" ] ; then
      runner_args="$runner_args -Dsonar.java.test.libraries=test/lib/**/*.jar"
    fi

    if [ -f $WORKDIR/$rev/test/jacoco.exec ] ; then
      runner_args="$runner_args -Dsonar.jacoco.reportPath=test/jacoco.exec"
    fi

    if [ -f $WORKDIR/$rev/test/jacocoIT.exec ] ; then
      runner_args="$runner_args -Dsonar.jacoco.itReportPath=test/jacocoIT.exec"
    fi

    if [ -d $WORKDIR/$rev/test/report ] ; then
      runner_args="$runner_args -Dsonar.junit.reportsPath=test/report"
    fi
  fi

  if [ -d "$WORKDIR/$rev/lib" ] && [ "$(ls -A $WORKDIR/$rev/lib)" ] ; then
#    runner_args="$runner_args -Dsonar.java.libraries=lib/**/*.jar"
    runner_args="$runner_args -Dsonar.java.libraries=lib/*.jar"
  fi

  # For sonarqube.com only
  #runner_args="$runner_args -Dsonar.login=$PROJECT_TOKEN"

  # Launch SonarQube Scanner (previously "runner")
  echo $RUNNER -e $runner_args
  $RUNNER -e $runner_args >>$log 2>&1 || exit 10

  # clean
#  if [ $rev -ge 413 ] ; then
#    ant -S $cleantargets
#  fi
  rm -Rf "$WORKDIR/$rev/build" "$WORKDIR/$rev/test/build" 
#  rm -Rf "$WORKDIR/$rev/test/report" "$WORKDIR/$rev/test/jacoco.exec"
done

Could be useful to you.

comment:7 Changed 5 years ago by Don-vip

Keywords:	svn git migration script added

comment:8 follow-ups: 9 10 Changed 5 years ago by michael2402

Hmm, adding apache commons to our git development is not a good idea I think.

Maby we should really go the two-way approach of providing one repository that can be built (with all jar files and svn:externals) and one that only contains the source we edited and cannot be built for old versions of JOSM.

The first one can then be used to e.g. find out when a bug was introduced if one wants to go though the versions (we could also keep SVN running for that, but git has the advantage that we do only need to maintain one software and that it can easily be downloaded for offline use / backups.

For the first one, we could simply use our github mirror script.

comment:9 in reply to: 8 Changed 5 years ago by anonymous

Replying to michael2402:

Hmm, adding apache commons to our git development is not a good idea I think.

Why is that? The point of this script is to have in our history all versions that are ready to be built (provided that you'll have proper JDK at hand) and moving on with our dependencies outside of our repository.

We have one repository (which is quite small) that fulfils all our needs.

comment:10 in reply to: 8 Changed 5 years ago by wiktorn

comment no 9 - wiktorn

comment:11 follow-up: 12 Changed 5 years ago by wiktorn

Description:	modified (diff)

With updated scripts I get only following errors:

grep -h error report/*err | sort | uniq -c
      1      [exec] Parser generated with 0 errors and 2 warnings.
     20     [javac] 1 error
      9     [javac] /io/josm-tests-builds/src/org/openstreetmap/josm/io/MyHttpURLConnection.java:11: error: unreported exception IOException; must be caught or declared to be thrown
      4     [javac] /io/josm-tests-builds/src/org/openstreetmap/josm/io/MyHttpURLConnection.java:12: error: unreported exception IOException; must be caught or declared to be thrown
      6     [javac] /io/josm-tests-builds/src/org/openstreetmap/josm/tools/MultiMap.java:14: error: name clash: put(A,B) in MultiMap and put(K,V) in HashMap have the same erasure, yet neither overrides the other
      1     [javac] /io/josm-tests-builds/src/org/openstreetmap/josm/tools/MultiMap.java:15: error: name clash: put(A,B) in MultiMap and put(K,V) in HashMap have the same erasure, yet neither overrides the other

22 failed builds out of 156, all done with OpenJDK 8. It looks to me as scripts are almost ready to go as they are now. As above, only compile target was tested.

Now the history of externals is trimmed to the dates of presence in JOSM. It means also that there is no history of externals before JOSM.

TODOs:

change the author of initial commit/removal of externals
change the timestamp of last commit of external to one second before the JOSM revision

comment:12 in reply to: 11 Changed 5 years ago by Don-vip

Replying to wiktorn:

22 failed builds out of 156, all done with OpenJDK 8.

I think that's why I used older JDKs to build old versions:

  # identify java version
  if [[ $rev -lt 3378 && ( $rev -lt 3094 || ( $rev -ge 3105 && ( $rev -lt 3321 || $rev -ge 3336 ) ) ) ]] ; then
    java="1.5"
    jdk=$JDK5_HOME
  elif [ $rev -lt 7001 ] ; then
    java="1.6"
    jdk=$JDK6_HOME
  elif [ $rev -lt 10580 ] ; then
    java="1.7"
    jdk=$JDK7_HOME
  else
    java="1.8"
    jdk=$JDK8_HOME
  fi

The JDK5 test is funny :)

comment:13 Changed 4 weeks ago by stoecker

Resolution:	→ worksforme
Status:	new → closed

Modify Ticket

Change Properties

Summary:
Type:		Priority:
Milestone:		Component:
Version:		Keywords:
Cc:	Set your email in Preferences

Action

leave as closed The owner will remain team.

change resolution as The resolution will be set.

reopen The resolution will be deleted.

Add Comment

Your email or username:

E-mail address and name can be saved in the Preferences.

You may use WikiFormatting here.

Attachments ↑ Description ↑

Note: See TracTickets for help on using tickets.

Download in other formats: