Opened 5 years ago
Closed 4 weeks ago
#16944 closed enhancement (worksforme)
svn -> git migration script
Reported by: | wiktorn | Owned by: | team |
---|---|---|---|
Priority: | normal | Milestone: | |
Component: | unspecified | Version: | |
Keywords: | hack-weekend-2018-10 svn git migration script | Cc: |
Description (last modified by )
Split to 4 scripts to make it easier to restart script on errors
- git_migrate-1-init.sh
#!/bin/sh set -e git init josm-tests cd josm-tests # create commit, to keep master branch clear git commit -m 'init' --allow-empty ### # define remotes ### git svn init --trunk=. --prefix=svn_jmapviewer -R jmapviewer https://svn.openstreetmap.org/applications/viewer/jmapviewer git svn init --trunk=. --prefix=svn_josm -R josm https://josm.openstreetmap.de/svn/trunk # two separate remotes for commons as this is 2x ~10 minutes instead of 2+ houres to fetch # git svn init --trunk=. --include-paths='commons/proper/jcs' --prefix=svn_commons_jcs/ -R svn_commons_jcs https://svn.apache.org/repos/asf/commons/proper/jcs/trunk/commons-jcs-core/ git svn init --trunk=. --prefix=svn_commons_jcs/ -R svn_commons_jcs https://svn.apache.org/repos/asf/commons/proper/jcs/trunk/commons-jcs-core/ # git svn init --trunk=. --prefix=svn_commons/ -R svn_commons https://svn.apache.org/repos/asf/ git svn init --trunk=trunk -b branches -t tags --prefix=svn_commons_logging/ -R svn_commons_logging https://svn.apache.org/repos/asf/commons/proper/logging/ git svn init --trunk=trunk -b branches -t tags --prefix=svn_commons_codec/ -R svn_commons_codec https://svn.apache.org/repos/asf/commons/proper/codec/ git svn init --trunk=trunk -b branches -t tags --prefix=svn_commons_bzip2/ -R svn_commons_bzip2 http://svn.apache.org/repos/asf/ant/core/ git svn init --trunk=. --prefix=svn_commons_bzip2_small/ -R svn_commons_bzip2_small http://svn.apache.org/repos/asf/ant/core/trunk/src/main/org/apache/tools/bzip2 git remote add commons_compress https://github.com/apache/commons-compress.git echo Done
- git_migrate-2-fetch.sh
#!/bin/sh set -e cd josm-tests ### # fetch from remotes ### git svn fetch jmapviewer git svn fetch josm # git svn fetch --log-window-size=100000 svn_commons git svn fetch --log-window-size=100000 svn_commons_jcs git svn fetch --log-window-size=100000 svn_commons_logging git svn fetch --log-window-size=100000 svn_commons_codec # git svn fetch --log-window-size=100000 svn_commons_bzip2 git svn fetch --log-window-size=100000 svn_commons_bzip2_small git fetch commons_compress echo Done
- git_migrate-3-prepare.sh
#!/bin/sh set -e cd josm-tests ### # Reorganize externals ### # TODO - for all - squash all commits before introduction to JOSM to one # remove files, on removal in JOSM # JMapViewer git branch -f svn_jmapviewer_rewrite remotes/svn_jmapviewertrunk git filter-branch -f --prune-empty --tree-filter ' if [ -d src/org/openstreetmap/gui/jmapviewer ] ; then git mv src src.new && find . -depth 1 | grep -v ^./src.new | grep -v ^./.git$ | xargs -r git rm -fr && mkdir -p src/org/openstreetmap/gui && git add src/org/openstreetmap/gui && git mv src.new/org/openstreetmap/gui/jmapviewer src/org/openstreetmap/gui && git rm -rf --ignore-unmatch src.new else find . -depth 1 | grep -v ^./.git$ | xargs -r git rm -fr fi' \ svn_jmapviewer_rewrite # Apache Commons Compress # Linearlize commons_compress history git branch -f svn_commons_compress_rewrite remotes/commons_compress/master # linearlize history git filter-branch -f --parent-filter 'cut -f 2,3 -d " "' svn_commons_compress_rewrite # Extract externals git filter-branch -f --prune-empty --tree-filter ' if [ -d src/main/java ] ; then git mv src src.new && find . -depth 1 | grep -v ^./src.new | grep -v ^./.git$ | xargs -r git rm -fr && git mv src.new/main/java src && git rm -rf --ignore-unmatch src.new else find . -depth 1 | grep -v ^./.git$ | xargs -r git rm -fr fi' \ svn_commons_compress_rewrite # Apache Commons JCS git branch -f svn_commons_jcstrunk_rewrite remotes/svn_commons_jcs/trunk # git filter-branch -f --prune-empty --tree-filter ' # if [ -d trunk/commons-jcs-core/src/main/java/ ] ; then # git rm -rf src ; # git mv trunk/commons-jcs-core/src/main/java src && # find . -depth 1 | # grep -v ^./src$ | grep -v ^./.git$ | # xargs -r git rm -fr # else # find . -depth 1 | # grep -v ^./.git$ | # xargs -r git rm -fr # fi' \ # svn_commons_jcstrunk_rewrite git filter-branch -f --prune-empty --tree-filter ' if [ -d src/main/java/ ] ; then git mv src src.new && find . -depth 1 | grep -v ^./src.new | grep -v ^./.git$ | xargs -r git rm -fr && git mv src.new/main/java src && git rm -rf --ignore-unmatch src.new else find . -depth 1 | grep -v ^./.git$ | xargs -r git rm -fr fi' \ svn_commons_jcstrunk_rewrite # Apache Commons Logging git branch -f svn_commons_loggingtrunk_rewrite remotes/svn_commons_logging/trunk git filter-branch -f --prune-empty --tree-filter ' if [ -d src/main/java ] ; then git mv src src.new && find . -depth 1 | grep -v ^./src.new | grep -v ^./.git$ | xargs -r git rm -fr && git mv src.new/main/java src && git rm -rf --ignore-unmatch src.new else find . -depth 1 | grep -v ^./.git$ | xargs -r git rm -fr fi' \ svn_commons_loggingtrunk_rewrite # TODO: Apache Codecs # svn_commons_codec git branch -f svn_commons_codec_rewrite remotes/svn_commons_codec/trunk git filter-branch -f --prune-empty --tree-filter ' if [ -d src/main/java ] ; then git mv src src.new && find . -depth 1 | grep -v ^./src.new | grep -v ^./.git$ | xargs -r git rm -fr && git mv src.new/main/java src && git rm -rf --ignore-unmatch src.new elif [ -d src/java ] ; then git mv src src.new && find . -depth 1 | grep -v ^./src.new | grep -v ^./.git$ | xargs -r git rm -fr && git mv src.new/java src && git rm -rf --ignore-unmatch src.new else find . -depth 1 | grep -v ^./.git$ | xargs -r git rm -fr fi' \ svn_commons_codec_rewrite git branch -f svn_commons_bzip2_rewrite remotes/svn_commons_bzip2_small/trunk git filter-branch -f --prune-empty --tree-filter ' mkdir -p src/org/apache/tools/bzip2 git mv *java src/org/apache/tools/bzip2 ' \ svn_commons_bzip2_rewrite
- git_migrate-4-adapt-history.sh
#!/bin/sh set -e cd josm-tests for i in svn_jmapviewer svn_commons_jcstrunk svn_commons_loggingtrunk svn_commons_codec svn_commons_compress svn_commons_bzip2 ; do git branch -f "${i}_clean" "${i}_rewrite" # git rebase $(git log --reverse --oneline --no-abbrev-commit "${i}_clean" | head -n 1 | cut -d ' ' -f 1) "${i}_clean" || exit 1 done git branch -f svn_commons_compress_compressors_clean svn_commons_compress_rewrite git branch -f svn_commons_compress_utils_clean svn_commons_compress_rewrite function josm_ver_to_timestamp { git log --grep "git-svn-id: https://josm.openstreetmap.de/svn/trunk@${1}" --pretty="format:%at" svn_josmtrunk } function first_before { BRANCH="${1}" TIMESTAMP="${2}" COMMIT_NO=$(( $( git log --pretty="format:%at %H" "${BRANCH}" | sort -n | awk -F ' ' "\$1 > ${TIMESTAMP} {print NR} {} " | head -n 1) - 1 )) git log --pretty="format:%at %H" ${BRANCH} | sort -n | awk -F ' ' "NR == ${COMMIT_NO} {print \$2}" } function first_after { BRANCH="${1}" TIMESTAMP="${2}" #COMMIT_NO=$( git log --pretty="format:%at %H" "${BRANCH}" | sort -n | awk -F ' ' "\$1 < ${TIMESTAMP} {print \$2} {} " | head -n 1) git log --pretty="format:%at %H" "${BRANCH}" | sort -n | awk -F ' ' "\$1 < ${TIMESTAMP} {print \$2} {} " | tail -n 1 #git log --pretty="format:%at %H" ${BRANCH} | sort -n | awk -F ' ' "NR == ${COMMIT_NO} {print \$2}" } function extract_timestamp { echo $1 | sed -E -e "s%[^0-9]*([0-9]*)[^0-9]*.*%\1%" } function truncate_from_to { BRANCH="$1" START_TS="$2" STOP_TS="$3" echo "Working on ${BRANCH}" if [ -z "${BRANCH}" ] ; then echo No branch provided exit 1; fi if [ -z "${START_TS}" ] ; then echo no STATRT_TS provided exit 1; fi COMMIT_ID=$(first_before "${BRANCH}" "${START_TS}") if [ -z "${COMMIT_ID}" ] ; then echo Unable to find commit on "${BRANCH}" before "${START_TS}" exit 1 fi export BRANCH export START_TS export COMMIT_ID git filter-branch -f --commit-filter ' if [ "${GIT_COMMIT}" == "${COMMIT_ID}" ] ; then GIT_AUTHOR_DATE="${START_TS}" git commit-tree -m "Initial import of ${BRANCH}" "$@" elif [ $(echo "${GIT_AUTHOR_DATE}" | sed -E -e "s%[^0-9]*([0-9]*)[^0-9]*.*%\1%" ) -lt "${START_TS}" ] ; then skip_commit "$@" else git commit-tree "$@" fi ' ${BRANCH} if [ -n "${STOP_TS}" ] ; then COMMIT_ID=$(first_after "${BRANCH}" "${STOP_TS}") export COMMIT_ID export STOP_TS git filter-branch -f --tree-filter ' if [ "${GIT_COMMIT}" == "${COMMIT_ID}" ]; then find . -depth 1 | grep -v ^./git$ | xargs -r git rm -rf fi ' --commit-filter ' if [ "${GIT_COMMIT}" == "${COMMIT_ID}" ] ; then GIT_AUTHOR_DATE="${STOP_TS}" git commit-tree -m "Remove ${BRANCH}" "$@" elif [ $(echo "${GIT_AUTHOR_DATE}" | sed -E -e "s%[^0-9]*([0-9]*)[^0-9]*.*%\1%" ) -gt "${STOP_TS}" ] ; then skip_commit "$@" else git commit-tree "$@" fi ' ${BRANCH} fi } ### # Clean compress_compressors ### git filter-branch -f --tree-filter ' find src/org/apache/commons/compress -depth 1 | grep -v compressors | xargs -r git rm -rf ' svn_commons_compress_compressors_clean git filter-branch -f --tree-filter ' find src/org/apache/commons/compress -depth 1 | grep -v utils | xargs -r git rm -rf ' svn_commons_compress_utils_clean truncate_from_to svn_commons_codec_clean $(josm_ver_to_timestamp 2796) $(josm_ver_to_timestamp 8149) truncate_from_to svn_commons_bzip2_clean $(josm_ver_to_timestamp 2796) $(josm_ver_to_timestamp 7867) truncate_from_to svn_commons_compress_compressors_clean $(josm_ver_to_timestamp 7867) $(josm_ver_to_timestamp 12499) truncate_from_to svn_commons_compress_utils_clean $(josm_ver_to_timestamp 11569) $(josm_ver_to_timestamp 12499) truncate_from_to svn_commons_compress_clean $(josm_ver_to_timestamp 12500) truncate_from_to svn_commons_jcstrunk_clean $(josm_ver_to_timestamp 8168) truncate_from_to svn_commons_loggingtrunk_clean $(josm_ver_to_timestamp 8168) # get history of svn externals: # svn propget -R svn:externals src/org/apache # then e.g. # svn log src/org/apache/commons # svn propget -r 7867 -R svn:externals src/org/apache # since 2796 codec - http://svn.apache.org/repos/asf/commons/proper/codec/trunk/src/java/org/apache/commons/codec # tools/bzip2 - http://svn.apache.org/repos/asf/ant/core/trunk/src/main/org/apache/tools/bzip2 # since 4460 - codec - http://svn.apache.org/repos/asf/commons/proper/codec/trunk/src/main/java/org/apache/commons/codec # since 7867 - tools/bzip2 -> compress -> http://svn.apache.org/repos/asf/commons/proper/compress/trunk/src/main/java/org/apache/commons/compress/compressors # since 8149 - no codec # since 8168 - jcs - http://svn.apache.org/repos/asf/commons/proper/jcs/trunk/commons-jcs-core/src/main/java/org/apache/commons/jcs # - logging - http://svn.apache.org/repos/asf/commons/proper/logging/trunk/src/main/java/org/apache/commons/logging logging # since 8698 - compress -> https://github.com/apache/commons-compress/trunk/src/main/java/org/apache/commons/compress/compressors # since 11569 - utils -> https://github.com/apache/commons-compress/trunk/src/main/java/org/apache/commons/compress/utils # since 12500 - compress -> https://github.com/apache/commons-compress/trunk/src/main/java/org/apache/commons/compress compress
- git_migrate-5-merge.sh
#!/bin/sh set -e cd josm-tests ### # Merge JOSM repository with externals, commits ordered by date ### # each git log needs echo at the end because first/last commit will be lost ( git log --pretty="format:%at %H %s" remotes/svn_josmtrunk ; echo git log --pretty="format:%at %H %s" svn_jmapviewer_clean ; echo git log --pretty="format:%at %H %s" svn_commons_jcstrunk_clean ; echo git log --pretty="format:%at %H %s" svn_commons_loggingtrunk_clean ; echo git log --pretty="format:%at %H %s" svn_commons_codec_clean ; echo git log --first-parent --pretty="format:%at %H %s" svn_commons_compress_clean ; echo git log --first-parent --pretty="format:%at %H %s" svn_commons_compress_compressors_clean; echo git log --first-parent --pretty="format:%at %H %s" svn_commons_compress_utils_clean ; echo git log --pretty="format:%at %H %s" svn_commons_bzip2_clean; echo ) | sort -n | cut -d ' ' -f2 | sed -e 's/^/p /' > ../rebase_config git config merge.renamelimit 5000 git checkout master git rebase --keep-empty -i $(git log --reverse --oneline --no-abbrev-commit | head -n 1 | cut -d ' ' -f 1) # helpful one-liner: git diff --name-only --diff-filter=U | xargs git checkout ... # remove empty commits git rebase $(git log --reverse --oneline --no-abbrev-commit | head -n 1 | cut -d ' ' -f 1) ### # Move jar files to Git LFS ### git lfs migrate import --include="*.jar" --include-ref=refs/heads/master git reflog expire --expire-unreachable=now --all git gc --prune=now
Attachments (0)
Change History (13)
comment:1 Changed 5 years ago by
comment:2 Changed 5 years ago by
Description: | modified (diff) |
---|
Split to 4 scripts. Apart from 1, all may be restarted.
I'll also try to check if JOSM actually builds from this (like every 100 commit or so).
comment:3 Changed 5 years ago by
Description: | modified (diff) |
---|
comment:4 Changed 5 years ago by
Description: | modified (diff) |
---|
comment:5 Changed 5 years ago by
I've run following script to check how does the old versions build:
#!/bin/bash for i in $(cd josm-tests-builds && git log --oneline --reverse --no-abbrev | awk '0 == NR % 100' | cut -d ' ' -f 1) ; do ( cd josm-tests-builds && git reset --hard && git clean -f git checkout $i && ant clean compile clean > ../report/${i}.err && mv ../report/${i}.err ../report/${i}.ok ) done
This gets every 100th commit. As GIT history starts before JOSM (I haven't squashed those commits, not sure if it's necessary), few first builds fail, as there is no build.xml.
There are also different failures. Probably due to changes between version that was used and JDK8:
- error: unreported exception IOException; must be caught or declared to be thrown
- error: name clash: put(A,B) in MultiMap and put(K,V) in HashMap have the same erasure, yet neither overrides the other
- error: cannot access JPEGDecodeParam
Missing externals:
- error: package org.apache.commons.codec.binary does not exist
- error: package org.apache.commons.compress.compressors.bzip2 does not exist
- error: package org.apache.commons.compress.compressors.xz does not exist
Problems in earlier versions of JCS:
- error: method does not override or implement a method from a supertype
- error: package org.apache.velocity does not exist
- error: package javax.servlet.http does not exist
Problems in logging:
- error: package org.apache.avalon.framework.logger does not exist
- error: package org.apache.log4j does not exist
- error: package org.apache.log does not exist
So the lesson here is, that externals should be squashed to the time they were introduced, and if they were removed - remove them also from our repo.
Regardless of the above, 20 builds succeeded :-)
comment:6 Changed 5 years ago by
When we reached commit 10.000 I re-run every past build to recreate SonarQube history.
I had two scripts:
<project name="josm" default="test-it" basedir="." xmlns:jacoco="antlib:org.jacoco.ant"> <path id="test.classpath"> <pathelement path="build"/> <pathelement path="test/build"/> <fileset dir="dist" erroronmissingdir="false"> <include name="josm-custom.jar"/> </fileset> <fileset dir="lib" erroronmissingdir="false"> <include name="**/*.jar"/> </fileset> <fileset dir="test/lib" erroronmissingdir="false"> <include name="**/*.jar"/> <exclude name="junit/**/*.jar"/> <exclude name="fest/**/*.jar"/> </fileset> <fileset dir="tools" erroronmissingdir="false"> <include name="findbugs/annotations.jar"/> <include name="groovy-all*.jar"/> </fileset> <fileset dir="/home/josm/auto/svn_josm/core/test/lib"> <include name="junit/**/*.jar"/> <include name="fest/**/*.jar"/> </fileset> </path> <target name="test"> <taskdef uri="antlib:org.jacoco.ant" resource="org/jacoco/ant/antlib.xml" classpath="/home/josm/auto/svn_josm/core/tools/jacocoant.jar" /> <jacoco:coverage destfile="test/jacoco.exec"> <junit printsummary="yes" fork="true" forkmode="once"> <jvmarg value="-Dfile.encoding=UTF-8"/> <sysproperty key="josm.home" value="test/config/unit-josm.home"/> <sysproperty key="josm.test.data" value="test/data"/> <sysproperty key="java.awt.headless" value="true"/> <classpath> <path refid="test.classpath"/> <pathelement path="test/config"/> </classpath> <formatter type="plain"/> <formatter type="xml"/> <batchtest fork="yes" todir="test/report"> <fileset dir="test" includes="org/**/*Test.java" erroronmissingdir="false"/> <fileset dir="test/unit" includes="org/**/*Test.java" erroronmissingdir="false"/> <fileset dir="test/functional" includes="org/**/*Test.java" erroronmissingdir="false"/> <fileset dir="test/performance" includes="org/**/*Test.java" erroronmissingdir="false"/> </batchtest> </junit> </jacoco:coverage> </target> <target name="test-it" depends="test"> <jacoco:coverage destfile="test/jacocoIT.exec"> <junit printsummary="yes" fork="true" forkmode="once"> <jvmarg value="-Dfile.encoding=UTF-8"/> <sysproperty key="josm.home" value="test/config/unit-josm.home"/> <sysproperty key="josm.test.data" value="test/data"/> <sysproperty key="java.awt.headless" value="true"/> <classpath> <path refid="test.classpath"/> <pathelement path="test/config"/> </classpath> <formatter type="plain"/> <formatter type="xml"/> <batchtest fork="yes" todir="test/report"> <fileset dir="test" includes="org/**/*TestIT.java" erroronmissingdir="false"/> <fileset dir="test/unit" includes="org/**/*TestIT.java" erroronmissingdir="false"/> <fileset dir="test/functional" includes="org/**/*TestIT.java" erroronmissingdir="false"/> <fileset dir="test/performance" includes="org/**/*TestIT.java" erroronmissingdir="false"/> </batchtest> </junit> </jacoco:coverage> </target> </project>
=> This custom build.xml used up-to-date tools stored outside the SVN repository
#!/bin/bash # ------------------------------------------------------------------------- # # JOSM Historical data feeder for SonarQube # # ------------------------------------------------------------------------- # # Feeds historical data of JOSM development to a SonarQube instance # # # # This script takes care of the following events in JOSM history # # https://josm.openstreetmap.de/changeset/109/josm : add build.xml # # https://josm.openstreetmap.de/changeset/111/josm : remove debug in build # # https://josm.openstreetmap.de/changeset/269/josm : use JUnit 4 # # https://josm.openstreetmap.de/changeset/321/josm : trunk directory # # https://josm.openstreetmap.de/changeset/413/josm : add debug in build # # https://josm.openstreetmap.de/changeset/1156/josm: remove unit tests # # https://josm.openstreetmap.de/changeset/1622/josm: add unit tests # # https://josm.openstreetmap.de/changeset/1751/josm: add test target # # https://josm.openstreetmap.de/changeset/1752/josm: add wrong env file # # https://josm.openstreetmap.de/changeset/3094/josm: wrong Java 6 commit # # https://josm.openstreetmap.de/changeset/3105/josm: revert Java 6 commit # # https://josm.openstreetmap.de/changeset/3321/josm: wrong Java 6 commit # # https://josm.openstreetmap.de/changeset/3336/josm: revert Java 6 commit # # https://josm.openstreetmap.de/changeset/3378/josm: switch to Java 6 # # https://josm.openstreetmap.de/changeset/7001/josm: switch to Java 7 # # https://josm.openstreetmap.de/changeset/9501/josm: unit/IT tests # # ------------------------------------------------------------------------- # OSM_USERNAME="xxx" OSM_PASSWORD="xxx" WORKDIR="/home/josm/team/Vincent/historic" RUNNER="/opt/sonar-scanner-2.5/bin/sonar-runner" JDK5_HOME="/opt/jdk1.5.0_22" JDK6_HOME="/opt/jdk1.6.0_45" JDK7_HOME="/opt/jdk1.7.0_80" JDK8_HOME="/usr/lib/jvm/java-8-openjdk-amd64" JDK9_HOME="/usr/lib/jvm/java-9-openjdk-amd64" JUNIT3="/usr/share/java/junit-3.8.2.jar" JUNIT4="/home/josm/auto/svn_josm/core/test/lib/junit/junit-4.12.jar:/home/josm/auto/svn_josm/core/test/lib/junit/hamcrest-core-1.3.jar" JACOCO="/home/josm/auto/svn_josm/core/tools/jacocoant.jar" HOST_URL="https://josm.openstreetmap.de/sonar" PROJECT_KEY="josm_releases" PROJECT_NAME="JOSM_Releases" #HOST_URL="https://sonarqube.com" #PROJECT_KEY="josm" #PROJECT_NAME="JOSM" #PROJECT_TOKEN="xxx" TRAC_URL="https://josm.openstreetmap.de" TRAC_USER="xxx" TRAC_PASSWORD="xxx" function manual_tests { rev=$1 java=$2 jdk=$3 libs=$4 rm -Rf "$WORKDIR/$rev/test/build" "$WORKDIR/$rev/test/report" "$WORKDIR/$rev/test/jacoco.exec" mkdir "$WORKDIR/$rev/test/build" if [ $rev -lt 269 ] ; then junit=$JUNIT3 javac=$JDK5_HOME/bin/javac else junit=$JUNIT4 fi if [ -d "$WORKDIR/$rev/test/unit" ] ; then testdir="$WORKDIR/$rev/test/unit" for d in "functional" "performance" ; do if [ -d "$WORKDIR/$rev/test/$d" ] ; then testdir="$testdir:$WORKDIR/$rev/test/$d" fi done else testdir="$WORKDIR/$rev/test" fi td="/home/josm/auto/svn_josm/core/test/lib" testlibs="$libs:$WORKDIR/$rev/lib/jfcunit.jar:$td/fest/MRJToolkitStubs-1.0.jar:$td/fest/debug-1.0.jar:$td/fest/fest-assert-1.0.jar:$td/fest/fest-reflect-1.1.jar:$td/fest/fest-swing-1.1.jar:$td/fest/fest-util-1.0.jar:$td/fest/jcip-annotations-1.0.jar:$td/unitils-core/unitils-core-3.4.2.jar:$td/unitils-core/ognl-2.6.9.jar:$td/unitils-core/commons-logging-1.1.jar:$td/unitils-core/commons-lang-2.3.jar:$td/unitils-core/commons-collections-3.2.jar:$WORKDIR/$rev/test/lib/equalsverifier-1.7.6.jar:$WORKDIR/$rev/test/lib/equalsverifier-2.0.2.jar:$WORKDIR/$rev/test/lib/equalsverifier-2.1.5.jar:$WORKDIR/$rev/tools/findbugs/annotations.jar" IFS=':' read -ra ADDR <<< "$testdir" for d in "${ADDR[@]}"; do for f in `find $d -name *Test.java` ; do $jdk/bin/javac -encoding UTF8 -source $java -target $java -d "$WORKDIR/$rev/test/build" -cp "$WORKDIR/$rev/build:$junit:$testdir:$testlibs" $f if [ ! $? -eq 0 ] && [ $rev -ge 1622 ] && [ $d != "$WORKDIR/$rev/test/functional" ] ; then echo "Unexpected compilation error" exit 100 fi done done mkdir "$WORKDIR/$rev/test/report" if [ -f "$WORKDIR/$rev/test/config/preferences.template.xml" ] ; then for fam in "unit" "functional" "performance" ; do if [ -d "$WORKDIR/$rev/test/config/$fam-josm.home" ] ; then sed "s/@OSM_USERNAME@/$OSM_USERNAME/g; s/@OSM_PASSWORD@/$OSM_PASSWORD/g" $WORKDIR/$rev/test/config/preferences.template.xml > $WORKDIR/$rev/test/config/$fam-josm.home/preferences.xml fi done fi ant -Dbasedir="$WORKDIR/$rev" -f "$WORKDIR/build_test.xml" | grep --line-buffered FAILED } # Almost all tested releases (not the very close ones) for rev in 14 25 35 40 43 53 61 100 166 285 802 869 898 929 970 1010 1065 1091 1097 1137 1178 1198 1212 1344 1387 1437 1460 1479 1504 1515 1529 1566 1607 1669 1788 1981 2221 2255 2552 2554 2561 3094 3208 3329 3376 3514 3592 3701 3751 3965 3966 4021 4064 4201 4223 4279 4399 4487 4550 4667 4878 5047 5181 5207 5210 5267 5315 5356 5485 5531 5576 5608 5697 5759 5836 5939 5990 6060 6115 6238 6388 6502 6766 6767 6891 6950 7000 7182 7287 7347 7480 7588 7643 7777 7906 7995 8109 8159 8279 8339 8491 8677 8800 8969 9060 9229 9329 9900 9979 10168 10301 10526 10786 10966 do baseurl="http://josm.openstreetmap.de/svn" if [ $rev -gt 321 ] ; then baseurl="$baseurl/trunk" fi log="$WORKDIR/$rev/log.txt" if [ -f $log ] ; then rm -f $log fi if [ ! -d $WORKDIR/$rev ] ; then svn co -q $baseurl@$rev $WORKDIR/$rev #|| exit 1 fi cd $WORKDIR/$rev || exit 2 date=`svn info | grep "Last Changed Date" | cut -d" " -f4` # replace svn:externals that do not exist anymore if [ -d "$WORKDIR/$rev/src/org/apache/commons" ] ; then codec=`svn propget svn:externals $WORKDIR/$rev/src/org/apache/commons | grep codec | cut -d" " -f2` if [ "$codec" = "http://svn.apache.org/repos/asf/commons/proper/codec/trunk/src/java/org/apache/commons/codec" ] ; then svn co -q http://svn.apache.org/repos/asf/commons/proper/codec/trunk/src/java/org/apache/commons/codec@1173286 $WORKDIR/$rev/src/org/apache/commons/codec || exit 2 fi if [ -d "$WORKDIR/$rev/src/org/apache/commons/compress" ] ; then compressors=`svn propget svn:externals $WORKDIR/$rev/src/org/apache/commons/compress | grep compressors | cut -d" " -f2` if [ "$compressors" = "http://svn.apache.org/repos/asf/commons/proper/compress/trunk/src/main/java/org/apache/commons/compress/compressors" ] ; then svn co -q https://github.com/apache/commons-compress/trunk/src/main/java/org/apache/commons/compress/compressors $WORKDIR/$rev/src/org/apache/commons/compress/compressors || exit 2 fi fi fi # checkout svn:externals at the correct revision for d in "data" "data_nodist" "eclipse" "images" "images_nodist" "lib" "linux" "macosx" "netbeans" "patches" "presets" "resources" "scripts" "src" "styles" "styles_nodist" "test" "tools" "windows" ; do if [ -d $WORKDIR/$rev/$d ] ; then find $WORKDIR/$rev/$d -name .svn -execdir svn update -q -r {$date} \; fi done # identify java version if [[ $rev -lt 3378 && ( $rev -lt 3094 || ( $rev -ge 3105 && ( $rev -lt 3321 || $rev -ge 3336 ) ) ) ]] ; then java="1.5" jdk=$JDK5_HOME elif [ $rev -lt 7001 ] ; then java="1.6" jdk=$JDK6_HOME elif [ $rev -lt 10580 ] ; then java="1.7" jdk=$JDK7_HOME else java="1.8" jdk=$JDK8_HOME fi # workaround for tests placed in source folders (r100) if [ -d "$WORKDIR/$rev/src/org/openstreetmap/josm/test" ] && [ ! -d "$WORKDIR/$rev/test" ] ; then mkdir -p "$WORKDIR/$rev/test/org/openstreetmap/josm" mv "$WORKDIR/$rev/src/org/openstreetmap/josm/test" "$WORKDIR/$rev/test/org/openstreetmap/josm/" fi # remove invalid unit test env file if [ -f "$WORKDIR/$rev/test/config/test-unit-env.properties" ] ; then rm -f "$WORKDIR/$rev/test/config/test-unit-env.properties" fi # compile and run tests export JAVACMD=$jdk/bin/java if [ $rev -ge 413 ] ; then # we have a build.xml with debug compilation if [ $rev -lt 9842 ] ; then # but we run tests manually to be sure to use jacoco cleantargets="clean" targets="dist" else cleantargets="clean test-clean" targets="dist test" fi ant -S $cleantargets $targets || exit 3 # we have unit tests until r1156, working fine in earlier revisions, then slowly abandoned, then restored in r1622, but without target until r1751, and jacoco latest change in 9842 if [[ $rev -lt 1156 || ( $rev -ge 1622 && $rev -lt 9842 ) ]] ; then manual_tests $rev $java $jdk $libs fi else # manual compilation, we don't have any build.xml, or a build without debug compilation libdir="$WORKDIR/$rev/lib" libs="$libdir/jdom.jar:$libdir/MinML2.jar:$libdir/metadata-extractor-2.3.1.jar:$libdir/gettext-commons-0.9.jar" rm -Rf "$WORKDIR/$rev/build" mkdir "$WORKDIR/$rev/build" $JDK5_HOME/bin/javac -encoding UTF8 -source $java -target $java -d "$WORKDIR/$rev/build" -cp "$WORKDIR/$rev/src:$libs" `find src/org/openstreetmap/josm -name Main*.java` || exit 4 if [ -f "$WORKDIR/$rev/src/org/openstreetmap/josm/data/conflict/Merger.java" ] ; then $JDK5_HOME/bin/javac -encoding UTF8 -source $java -target $java -d "$WORKDIR/$rev/build" -cp "$WORKDIR/$rev/src:$libs" "$WORKDIR/$rev/src/org/openstreetmap/josm/data/conflict/Merger.java" || exit 5 fi if [ -d "$WORKDIR/$rev/test" ] ; then # manual tests manual_tests $rev $java $jdk $libs fi fi unset JAVACMD # identify JOSM version if [ $rev -eq 14 ] ; then version="1.0b1" elif [ $rev -eq 25 ] ; then version="1.0b2" elif [ $rev -eq 35 ] ; then version="1.0rc1" elif [ $rev -eq 40 ] ; then version="1.0rc2" elif [ $rev -eq 43 ] ; then version="1.0" elif [ $rev -eq 53 ] ; then version="1.1" elif [ $rev -eq 61 ] ; then version="1.2" elif [ $rev -eq 100 ] ; then version="1.3" elif [ $rev -eq 166 ] ; then version="1.4" elif [ $rev -eq 285 ] ; then version="1.5" else version=$rev fi echo "Analyzing r$rev - $date - $version - java $java" # List of source folders sources="src" for d in "data" "images" "linux" "macosx" "presets" "resources" "scripts" "styles" "windows" ; do if [ -d $WORKDIR/$rev/$d ] ; then sources="$sources,$d" fi done # SonarQube analysis parameters runner_args="-Dsonar.host.url=$HOST_URL -Dsonar.projectKey=$PROJECT_KEY -Dsonar.projectName=$PROJECT_NAME -Dsonar.projectVersion=$version -Dsonar.projectDate=$date -Dsonar.sources=$sources -Dsonar.sourceEncoding=UTF-8 -Dsonar.java.source=$java -Dsonar.java.binaries=build -Dsonar.scm.url=scm:$baseurl -Dsonar.findbugs.timeout=1200000 -Dsonar.trac.url=$TRAC_URL -Dsonar.trac.username.secured=$TRAC_USER -Dsonar.trac.password.secured=$TRAC_PASSWORD -Dsonar.links.homepage=$TRAC_URL -Dsonar.links.ci=$TRAC_URL/jenkins -Dsonar.links.issue=$TRAC_URL/report -Dsonar.links.scm=$TRAC_URL/browser/josm/trunk -Dsonar.links.scm_dev=https://josm.openstreetmap.de/svn/trunk -Djava_version=${java/1./} -Dsonar.inclusions=src/org/openstreetmap/josm/**/*.java,**/*.groovy,**/*.xml,**/*.xsd,**/*.css,**/*.osm,**/*.mapcss,**/*.properties,**/*.json,**/*.js,**/*.cfg -Dsonar.exclusions=src/org/openstreetmap/josm/gui/mappaint/mapcss/parsergen/**/*.java,src/org/openstreetmap/josm/data/imagery/types/*.java,data/overpass-turbo-ffs.js,data/overpass-wizard.js,data/validator/opening_hours.js,macosx/JOSM.app/Contents/Info.plist_template.xml -Dsonar.test.inclusions=test/**/*.java,test/**/*.groovy" # TODO does not work :'( # -Djava_version=${java/1./} if [ -d "$WORKDIR/$rev/test" ] ; then tests="" for t in "unit" "functional" "performance" ; do if [ -d "$WORKDIR/$rev/test/$t" ] ; then if [ ! -z "$tests" ] ; then tests="$tests," fi tests="${tests}test/$t" fi done if [ -z "$tests" ] ; then tests="test" fi runner_args="$runner_args -Dsonar.tests=$tests" if [ -d "$WORKDIR/$rev/test/lib" ] ; then runner_args="$runner_args -Dsonar.java.test.libraries=test/lib/**/*.jar" fi if [ -f $WORKDIR/$rev/test/jacoco.exec ] ; then runner_args="$runner_args -Dsonar.jacoco.reportPath=test/jacoco.exec" fi if [ -f $WORKDIR/$rev/test/jacocoIT.exec ] ; then runner_args="$runner_args -Dsonar.jacoco.itReportPath=test/jacocoIT.exec" fi if [ -d $WORKDIR/$rev/test/report ] ; then runner_args="$runner_args -Dsonar.junit.reportsPath=test/report" fi fi if [ -d "$WORKDIR/$rev/lib" ] && [ "$(ls -A $WORKDIR/$rev/lib)" ] ; then # runner_args="$runner_args -Dsonar.java.libraries=lib/**/*.jar" runner_args="$runner_args -Dsonar.java.libraries=lib/*.jar" fi # For sonarqube.com only #runner_args="$runner_args -Dsonar.login=$PROJECT_TOKEN" # Launch SonarQube Scanner (previously "runner") echo $RUNNER -e $runner_args $RUNNER -e $runner_args >>$log 2>&1 || exit 10 # clean # if [ $rev -ge 413 ] ; then # ant -S $cleantargets # fi rm -Rf "$WORKDIR/$rev/build" "$WORKDIR/$rev/test/build" # rm -Rf "$WORKDIR/$rev/test/report" "$WORKDIR/$rev/test/jacoco.exec" done
Could be useful to you.
comment:7 Changed 5 years ago by
Keywords: | svn git migration script added |
---|
comment:8 follow-ups: 9 10 Changed 5 years ago by
Hmm, adding apache commons to our git development is not a good idea I think.
Maby we should really go the two-way approach of providing one repository that can be built (with all jar files and svn:externals) and one that only contains the source we edited and cannot be built for old versions of JOSM.
The first one can then be used to e.g. find out when a bug was introduced if one wants to go though the versions (we could also keep SVN running for that, but git has the advantage that we do only need to maintain one software and that it can easily be downloaded for offline use / backups.
For the first one, we could simply use our github mirror script.
comment:9 Changed 5 years ago by
Replying to michael2402:
Hmm, adding apache commons to our git development is not a good idea I think.
Why is that? The point of this script is to have in our history all versions that are ready to be built (provided that you'll have proper JDK at hand) and moving on with our dependencies outside of our repository.
We have one repository (which is quite small) that fulfils all our needs.
comment:11 follow-up: 12 Changed 5 years ago by
Description: | modified (diff) |
---|
With updated scripts I get only following errors:
grep -h error report/*err | sort | uniq -c 1 [exec] Parser generated with 0 errors and 2 warnings. 20 [javac] 1 error 9 [javac] /io/josm-tests-builds/src/org/openstreetmap/josm/io/MyHttpURLConnection.java:11: error: unreported exception IOException; must be caught or declared to be thrown 4 [javac] /io/josm-tests-builds/src/org/openstreetmap/josm/io/MyHttpURLConnection.java:12: error: unreported exception IOException; must be caught or declared to be thrown 6 [javac] /io/josm-tests-builds/src/org/openstreetmap/josm/tools/MultiMap.java:14: error: name clash: put(A,B) in MultiMap and put(K,V) in HashMap have the same erasure, yet neither overrides the other 1 [javac] /io/josm-tests-builds/src/org/openstreetmap/josm/tools/MultiMap.java:15: error: name clash: put(A,B) in MultiMap and put(K,V) in HashMap have the same erasure, yet neither overrides the other
22 failed builds out of 156, all done with OpenJDK 8. It looks to me as scripts are almost ready to go as they are now. As above, only compile target was tested.
Now the history of externals is trimmed to the dates of presence in JOSM. It means also that there is no history of externals before JOSM.
TODOs:
- change the author of initial commit/removal of externals
- change the timestamp of last commit of external to one second before the JOSM revision
comment:12 Changed 5 years ago by
Replying to wiktorn:
22 failed builds out of 156, all done with OpenJDK 8.
I think that's why I used older JDKs to build old versions:
# identify java version if [[ $rev -lt 3378 && ( $rev -lt 3094 || ( $rev -ge 3105 && ( $rev -lt 3321 || $rev -ge 3336 ) ) ) ]] ; then java="1.5" jdk=$JDK5_HOME elif [ $rev -lt 7001 ] ; then java="1.6" jdk=$JDK6_HOME elif [ $rev -lt 10580 ] ; then java="1.7" jdk=$JDK7_HOME else java="1.8" jdk=$JDK8_HOME fi
The JDK5 test is funny :)
comment:13 Changed 4 weeks ago by
Resolution: | → worksforme |
---|---|
Status: | new → closed |
This script merges all (?) our externals into one git, ordering commits by date. Need some work on Apache Commons Compress, as date ordering is not always what should be.
Result of this script is downloadable from: http://gitlab.vink.pl/w/josm-tests.git
Repository has size of: 157MB
LFS objects: 918MB
Full clone (including current version of jar files): 233MB