Changes between Version 4 and Version 11 of Ticket #16944


Ignore:
Timestamp:
2018-11-13T07:59:39+01:00 (7 years ago)
Author:
wiktorn
Comment:

With updated scripts I get only following errors: 

grep -h error report/*err | sort | uniq -c
      1      [exec] Parser generated with 0 errors and 2 warnings.
     20     [javac] 1 error
      9     [javac] /io/josm-tests-builds/src/org/openstreetmap/josm/io/MyHttpURLConnection.java:11: error: unreported exception IOException; must be caught or declared to be thrown
      4     [javac] /io/josm-tests-builds/src/org/openstreetmap/josm/io/MyHttpURLConnection.java:12: error: unreported exception IOException; must be caught or declared to be thrown
      6     [javac] /io/josm-tests-builds/src/org/openstreetmap/josm/tools/MultiMap.java:14: error: name clash: put(A,B) in MultiMap and put(K,V) in HashMap have the same erasure, yet neither overrides the other
      1     [javac] /io/josm-tests-builds/src/org/openstreetmap/josm/tools/MultiMap.java:15: error: name clash: put(A,B) in MultiMap and put(K,V) in HashMap have the same erasure, yet neither overrides the other

22 failed builds out of 156, all done with OpenJDK 8. It looks to me as scripts are almost ready to go as they are now. As above, only compile target was tested.

Now the history of externals is trimmed to the dates of presence in JOSM. It means also that there is no history of externals before JOSM.

TODOs:

  • change the author of initial commit/removal of externals
  • change the timestamp of last commit of external to one second before the JOSM revision

Legend:

Unmodified
Added
Removed
Modified

  • Ticket #16944

    • Property Keywords svn git migration script added

  • Ticket #16944 – Description

    v4 v11  
    1818git svn init --trunk=. --prefix=svn_josm -R josm https://josm.openstreetmap.de/svn/trunk
    1919# two separate remotes for commons as this is 2x ~10 minutes instead of 2+ houres to fetch
    20 git svn init --trunk=. --include-paths='commons/proper/jcs/trunk/.*src/main/java/org/apache/commons/jcs' --prefix=svn_commons_jcs -R svn_commons_jcs https://svn.apache.org/repos/asf/commons/proper/jcs/
    21 git svn init --trunk=. --include-paths='commons/proper/logging/trunk/src/main/java/org/apache/commons/logging' --prefix=svn_commons_logging -R svn_commons_logging https://svn.apache.org/repos/asf/commons/proper/logging/trunk/
     20# git svn init --trunk=. --include-paths='commons/proper/jcs' --prefix=svn_commons_jcs/ -R svn_commons_jcs https://svn.apache.org/repos/asf/commons/proper/jcs/trunk/commons-jcs-core/
     21git svn init --trunk=. --prefix=svn_commons_jcs/ -R svn_commons_jcs https://svn.apache.org/repos/asf/commons/proper/jcs/trunk/commons-jcs-core/
     22# git svn init --trunk=. --prefix=svn_commons/ -R svn_commons https://svn.apache.org/repos/asf/
     23git svn init --trunk=trunk -b branches -t tags --prefix=svn_commons_logging/ -R svn_commons_logging https://svn.apache.org/repos/asf/commons/proper/logging/
     24git svn init --trunk=trunk -b branches -t tags --prefix=svn_commons_codec/ -R svn_commons_codec https://svn.apache.org/repos/asf/commons/proper/codec/
     25git svn init --trunk=trunk -b branches -t tags --prefix=svn_commons_bzip2/ -R svn_commons_bzip2 http://svn.apache.org/repos/asf/ant/core/
     26
     27git svn init --trunk=. --prefix=svn_commons_bzip2_small/ -R svn_commons_bzip2_small http://svn.apache.org/repos/asf/ant/core/trunk/src/main/org/apache/tools/bzip2
     28
    2229git remote add commons_compress https://github.com/apache/commons-compress.git
    2330
    2431echo Done
    25 }}}
     32
     33}}}
     34
     35
    2636* git_migrate-2-fetch.sh
    2737{{{#!bash
    … …  
    3646git svn fetch jmapviewer
    3747git svn fetch josm
     48# git svn fetch --log-window-size=100000 svn_commons
    3849git svn fetch --log-window-size=100000 svn_commons_jcs
    3950git svn fetch --log-window-size=100000 svn_commons_logging
    40 git svn fetch svn_commons_jcs
    41 git svn fetch svn_commons_logging
     51git svn fetch --log-window-size=100000 svn_commons_codec
     52# git svn fetch --log-window-size=100000 svn_commons_bzip2
     53git svn fetch --log-window-size=100000 svn_commons_bzip2_small
    4254git fetch commons_compress
    4355
    4456echo Done
    4557}}}
     58
     59
    4660* git_migrate-3-prepare.sh
    4761{{{#!bash
    … …  
    5468# Reorganize externals
    5569###
     70# TODO - for all - squash all commits before introduction to JOSM to one
     71# remove files, on removal in JOSM
    5672
    5773# JMapViewer
    5874git branch -f svn_jmapviewer_rewrite remotes/svn_jmapviewertrunk
    59 git filter-branch -f --tree-filter '
     75git filter-branch -f --prune-empty --tree-filter '
    6076  if [ -d src/org/openstreetmap/gui/jmapviewer ] ; then
    6177      git mv src src.new &&
    … …  
    7793# Linearlize commons_compress history
    7894git branch -f svn_commons_compress_rewrite remotes/commons_compress/master
     95# linearlize history
    7996git filter-branch -f --parent-filter 'cut -f 2,3 -d " "' svn_commons_compress_rewrite
    8097# Extract externals
     98git filter-branch -f --prune-empty --tree-filter '
     99  if [ -d src/main/java ] ; then
     100      git mv src src.new &&
     101      find . -depth 1 |
     102      grep -v ^./src.new | grep -v ^./.git$ |
     103      xargs -r git rm -fr &&
     104      git mv src.new/main/java src &&
     105      git rm -rf --ignore-unmatch src.new
     106  else
     107      find . -depth 1 |
     108      grep -v ^./.git$ |
     109      xargs -r git rm -fr
     110  fi' \
     111  svn_commons_compress_rewrite
     112
     113
     114# Apache Commons JCS
     115git branch -f svn_commons_jcstrunk_rewrite remotes/svn_commons_jcs/trunk
     116# git filter-branch -f --prune-empty --tree-filter '
     117#   if [ -d trunk/commons-jcs-core/src/main/java/ ] ; then
     118#       git rm -rf src ;
     119#       git mv trunk/commons-jcs-core/src/main/java src &&
     120#       find . -depth 1 |
     121#       grep -v ^./src$ | grep -v ^./.git$ |
     122#       xargs -r git rm -fr
     123#   else
     124#       find . -depth 1 |
     125#       grep -v ^./.git$ |
     126#       xargs -r git rm -fr
     127#   fi' \
     128#   svn_commons_jcstrunk_rewrite
     129git filter-branch -f --prune-empty --tree-filter '
     130  if [ -d src/main/java/ ] ; then
     131      git mv src src.new &&
     132      find . -depth 1 |
     133      grep -v ^./src.new | grep -v ^./.git$ |
     134      xargs -r git rm -fr &&
     135      git mv src.new/main/java src &&
     136      git rm -rf --ignore-unmatch src.new
     137  else
     138      find . -depth 1 |
     139      grep -v ^./.git$ |
     140      xargs -r git rm -fr
     141  fi' \
     142  svn_commons_jcstrunk_rewrite
     143
     144# Apache Commons Logging
     145git branch -f svn_commons_loggingtrunk_rewrite remotes/svn_commons_logging/trunk
     146git filter-branch -f --prune-empty --tree-filter '
     147  if [ -d src/main/java ] ; then
     148      git mv src src.new &&
     149      find . -depth 1 |
     150      grep -v ^./src.new | grep -v ^./.git$ |
     151      xargs -r git rm -fr &&
     152      git mv src.new/main/java src &&
     153      git rm -rf --ignore-unmatch src.new
     154  else
     155      find . -depth 1 |
     156      grep -v ^./.git$ |
     157      xargs -r git rm -fr
     158  fi' \
     159  svn_commons_loggingtrunk_rewrite
     160
     161# TODO: Apache Codecs
     162# svn_commons_codec
     163git branch -f svn_commons_codec_rewrite remotes/svn_commons_codec/trunk
     164git filter-branch -f --prune-empty --tree-filter '
     165  if [ -d src/main/java ] ; then
     166      git mv src src.new &&
     167      find . -depth 1 |
     168      grep -v ^./src.new | grep -v ^./.git$ |
     169      xargs -r git rm -fr &&
     170      git mv src.new/main/java src &&
     171      git rm -rf --ignore-unmatch src.new
     172  elif [ -d src/java ] ; then
     173      git mv src src.new &&
     174      find . -depth 1 |
     175      grep -v ^./src.new | grep -v ^./.git$ |
     176      xargs -r git rm -fr &&
     177      git mv src.new/java src &&
     178      git rm -rf --ignore-unmatch src.new
     179  else
     180      find . -depth 1 |
     181      grep -v ^./.git$ |
     182      xargs -r git rm -fr
     183  fi' \
     184  svn_commons_codec_rewrite
     185
     186git branch -f svn_commons_bzip2_rewrite remotes/svn_commons_bzip2_small/trunk
     187git filter-branch -f --prune-empty --tree-filter '
     188  mkdir -p src/org/apache/tools/bzip2
     189  git mv *java src/org/apache/tools/bzip2
     190  ' \
     191  svn_commons_bzip2_rewrite
     192
     193}}}
     194
     195
     196* git_migrate-4-adapt-history.sh
     197{{{#!bash
     198#!/bin/sh
     199
     200set -e
     201
     202cd josm-tests
     203
     204for i in svn_jmapviewer svn_commons_jcstrunk svn_commons_loggingtrunk svn_commons_codec svn_commons_compress svn_commons_bzip2 ; do
     205  git branch -f "${i}_clean" "${i}_rewrite"
     206  # git rebase $(git log --reverse --oneline --no-abbrev-commit "${i}_clean" | head -n 1 | cut -d ' ' -f 1) "${i}_clean" || exit 1
     207done
     208
     209git branch -f svn_commons_compress_compressors_clean svn_commons_compress_rewrite
     210git branch -f svn_commons_compress_utils_clean svn_commons_compress_rewrite
     211
     212
     213function josm_ver_to_timestamp {
     214  git log --grep "git-svn-id: https://josm.openstreetmap.de/svn/trunk@${1}" --pretty="format:%at" svn_josmtrunk
     215}
     216
     217function first_before {
     218  BRANCH="${1}"
     219  TIMESTAMP="${2}"
     220  COMMIT_NO=$(( $( git log --pretty="format:%at %H" "${BRANCH}" | sort -n |  awk -F ' ' "\$1 > ${TIMESTAMP} {print NR} {} " | head -n 1) - 1 ))
     221  git log --pretty="format:%at %H" ${BRANCH} | sort -n | awk -F ' ' "NR == ${COMMIT_NO} {print \$2}"
     222}
     223
     224function first_after {
     225  BRANCH="${1}"
     226  TIMESTAMP="${2}"
     227  #COMMIT_NO=$( git log --pretty="format:%at %H" "${BRANCH}" | sort -n |  awk -F ' ' "\$1 < ${TIMESTAMP} {print \$2} {} " | head -n 1)
     228  git log --pretty="format:%at %H" "${BRANCH}" | sort -n |  awk -F ' ' "\$1 < ${TIMESTAMP} {print \$2} {} " | tail -n 1
     229  #git log --pretty="format:%at %H" ${BRANCH} | sort -n | awk -F ' ' "NR == ${COMMIT_NO} {print \$2}"
     230}
     231
     232function extract_timestamp {
     233    echo $1 |  sed -E -e "s%[^0-9]*([0-9]*)[^0-9]*.*%\1%"
     234}
     235
     236function truncate_from_to {
     237    BRANCH="$1"
     238    START_TS="$2"
     239    STOP_TS="$3"
     240    echo "Working on ${BRANCH}"
     241    if [ -z "${BRANCH}" ] ; then
     242         echo No branch provided
     243         exit 1;
     244    fi
     245    if [ -z "${START_TS}" ] ; then
     246         echo no STATRT_TS provided
     247         exit 1;
     248    fi
     249
     250    COMMIT_ID=$(first_before "${BRANCH}" "${START_TS}")
     251
     252    if [ -z "${COMMIT_ID}" ] ; then
     253        echo Unable to find commit on "${BRANCH}" before "${START_TS}"
     254        exit 1
     255    fi
     256    export BRANCH
     257    export START_TS
     258    export COMMIT_ID
     259    git filter-branch -f --commit-filter '
     260      if [ "${GIT_COMMIT}" == "${COMMIT_ID}" ] ; then
     261        GIT_AUTHOR_DATE="${START_TS}" git commit-tree -m "Initial import of ${BRANCH}" "$@"
     262      elif [ $(echo "${GIT_AUTHOR_DATE}" | sed -E -e "s%[^0-9]*([0-9]*)[^0-9]*.*%\1%" )  -lt "${START_TS}" ] ; then
     263        skip_commit "$@"
     264      else
     265        git commit-tree "$@"
     266      fi
     267    ' ${BRANCH}
     268
     269
     270    if [ -n "${STOP_TS}" ] ; then
     271        COMMIT_ID=$(first_after "${BRANCH}" "${STOP_TS}")
     272        export COMMIT_ID
     273        export STOP_TS
     274        git filter-branch -f --tree-filter '
     275          if [ "${GIT_COMMIT}" == "${COMMIT_ID}" ]; then
     276             find . -depth 1 | grep -v ^./git$ | xargs -r git rm -rf
     277          fi
     278          ' --commit-filter '
     279          if [ "${GIT_COMMIT}" == "${COMMIT_ID}" ] ; then
     280            GIT_AUTHOR_DATE="${STOP_TS}" git commit-tree -m "Remove ${BRANCH}" "$@"
     281          elif [ $(echo "${GIT_AUTHOR_DATE}" | sed -E -e "s%[^0-9]*([0-9]*)[^0-9]*.*%\1%" ) -gt "${STOP_TS}" ] ; then
     282              skip_commit "$@"
     283          else
     284              git commit-tree "$@"
     285          fi
     286        ' ${BRANCH}
     287    fi
     288}
     289
     290###
     291# Clean compress_compressors
     292###
    81293git filter-branch -f --tree-filter '
    82   if [ -d src/main/java ] ; then
    83       git mv src src.new &&
    84       find . -depth 1 |
    85       grep -v ^./src.new | grep -v ^./.git$ |
    86       xargs -r git rm -fr &&
    87       git mv src.new/main/java src &&
    88       git rm -rf --ignore-unmatch src.new
    89   else
    90       find . -depth 1 |
    91       grep -v ^./.git$ |
    92       xargs -r git rm -fr
    93   fi' \
    94   svn_commons_compress_rewrite
    95 
    96 
    97 # Apache Commons JCS
    98 git branch -f svn_commons_jcstrunk_rewrite remotes/svn_commons_jcstrunk
     294  find src/org/apache/commons/compress -depth 1 | grep -v compressors | xargs -r git rm -rf
     295' svn_commons_compress_compressors_clean
     296
    99297git filter-branch -f --tree-filter '
    100   if [ -d trunk/commons-jcs-core/src/main/java/ ] ; then
    101       git rm -rf src ;
    102       git mv trunk/commons-jcs-core/src/main/java src &&
    103       find . -depth 1 |
    104       grep -v ^./src$ | grep -v ^./.git$ |
    105       xargs -r git rm -fr
    106   else
    107       find . -depth 1 |
    108       grep -v ^./.git$ |
    109       xargs -r git rm -fr
    110   fi' \
    111   svn_commons_jcstrunk_rewrite
    112 
    113 # Apache Commons Logging
    114 git branch -f svn_commons_loggingtrunk_rewrite remotes/svn_commons_loggingtrunk
    115 git filter-branch -f --tree-filter '
    116   if [ -d src/main/java ] ; then
    117       git mv src src.new &&
    118       find . -depth 1 |
    119       grep -v ^./src.new | grep -v ^./.git$ |
    120       xargs -r git rm -fr &&
    121       git mv src.new/main/java src &&
    122       git rm -rf --ignore-unmatch src.new
    123   else
    124       find . -depth 1 |
    125       grep -v ^./.git$ |
    126       xargs -r git rm -fr
    127   fi' \
    128   svn_commons_loggingtrunk_rewrite
    129 
    130 ###
    131 # Remove empty commits
    132 ###
    133 
    134 for i in svn_jmapviewer svn_commons_jcstrunk svn_commons_loggingtrunk ; do
    135   git branch -f "${i}_clean" "${i}_rewrite"
    136   git rebase $(git log --reverse --oneline --no-abbrev-commit "${i}_clean" | head -n 1 | cut -d ' ' -f 1) "${i}_clean" || exit 1
    137 done
    138 }}}
    139 * git_migrate-4-merge.sh
     298  find src/org/apache/commons/compress -depth 1 | grep -v utils | xargs -r git rm -rf
     299' svn_commons_compress_utils_clean
     300
     301truncate_from_to svn_commons_codec_clean $(josm_ver_to_timestamp 2796) $(josm_ver_to_timestamp 8149)
     302truncate_from_to svn_commons_bzip2_clean $(josm_ver_to_timestamp 2796) $(josm_ver_to_timestamp 7867)
     303truncate_from_to svn_commons_compress_compressors_clean $(josm_ver_to_timestamp 7867) $(josm_ver_to_timestamp 12499)
     304truncate_from_to svn_commons_compress_utils_clean $(josm_ver_to_timestamp 11569) $(josm_ver_to_timestamp 12499)
     305truncate_from_to svn_commons_compress_clean $(josm_ver_to_timestamp 12500)
     306truncate_from_to svn_commons_jcstrunk_clean $(josm_ver_to_timestamp 8168)
     307truncate_from_to svn_commons_loggingtrunk_clean $(josm_ver_to_timestamp 8168)
     308
     309# get history of svn externals:
     310# svn propget -R svn:externals src/org/apache
     311# then e.g.
     312# svn log src/org/apache/commons
     313# svn propget -r 7867 -R svn:externals src/org/apache
     314#  since 2796 codec - http://svn.apache.org/repos/asf/commons/proper/codec/trunk/src/java/org/apache/commons/codec
     315#             tools/bzip2       - http://svn.apache.org/repos/asf/ant/core/trunk/src/main/org/apache/tools/bzip2
     316#  since 4460 - codec - http://svn.apache.org/repos/asf/commons/proper/codec/trunk/src/main/java/org/apache/commons/codec
     317#  since 7867 - tools/bzip2 -> compress ->  http://svn.apache.org/repos/asf/commons/proper/compress/trunk/src/main/java/org/apache/commons/compress/compressors
     318# since 8149 - no codec
     319# since 8168 - jcs - http://svn.apache.org/repos/asf/commons/proper/jcs/trunk/commons-jcs-core/src/main/java/org/apache/commons/jcs
     320#            - logging - http://svn.apache.org/repos/asf/commons/proper/logging/trunk/src/main/java/org/apache/commons/logging logging
     321# since 8698 - compress ->  https://github.com/apache/commons-compress/trunk/src/main/java/org/apache/commons/compress/compressors
     322# since 11569 - utils -> https://github.com/apache/commons-compress/trunk/src/main/java/org/apache/commons/compress/utils
     323# since 12500 - compress -> https://github.com/apache/commons-compress/trunk/src/main/java/org/apache/commons/compress compress
     324}}}
     325
     326* git_migrate-5-merge.sh
    140327{{{#!bash
    141328#!/bin/sh
    … …  
    149336###
    150337
     338# each git log needs echo at the end because first/last commit will be lost
    151339(
    152   git log --pretty="format:%at %H %s" remotes/svn_josmtrunk
    153   git log --pretty="format:%at %H %s" svn_jmapviewer_rewrite
    154   git log --first-parent --pretty="format:%at %H %s" svn_commons_compress_clean
    155   git log --pretty="format:%at %H %s" svn_commons_jcstrunk_clean
    156   git log --pretty="format:%at %H %s" svn_commons_loggingtrunk_clean
     340  git log --pretty="format:%at %H %s" remotes/svn_josmtrunk ; echo
     341  git log --pretty="format:%at %H %s" svn_jmapviewer_clean ; echo
     342  git log --pretty="format:%at %H %s" svn_commons_jcstrunk_clean ; echo
     343  git log --pretty="format:%at %H %s" svn_commons_loggingtrunk_clean ; echo
     344  git log --pretty="format:%at %H %s" svn_commons_codec_clean ; echo
     345  git log --first-parent --pretty="format:%at %H %s" svn_commons_compress_clean ; echo
     346  git log --first-parent --pretty="format:%at %H %s" svn_commons_compress_compressors_clean; echo
     347  git log --first-parent --pretty="format:%at %H %s" svn_commons_compress_utils_clean ; echo
     348  git log --pretty="format:%at %H %s" svn_commons_bzip2_clean; echo
    157349) | sort -n | cut -d ' ' -f2 | sed -e 's/^/p /' > ../rebase_config
    158350
    … …  
    172364git reflog expire --expire-unreachable=now --all
    173365git gc --prune=now
    174 
    175 }}}
     366}}}