Modify

Opened 3 years ago

Closed 3 years ago

Last modified 3 years ago

#20146 closed enhancement (fixed)

Trigger macOS re-build on github when changeset marked at tested

Reported by: Stereo Owned by: stoecker
Priority: normal Milestone: 21.02
Component: Installer MacOS Version:
Keywords: macos Cc: Don-vip

Description

This is a work in progress, intended to run on the josm server when releasing a new 'tested' version. Instead of triggering a new macOS build locally like in the past, this finds the github action workflow that contained git-svn-id: https://josm.openstreetmap.de/svn/trunk@$josm_revision in its commit message.

When a git action run is triggered, it fetches https://josm.openstreetmap.de/tested. If /tested has been properly updated before triggering the rebuild, it will build as a release instead of a pre-release, like https://github.com/grischard/josm/releases/tag/17329-tested

The shell script requires jq to parse the github json.

I've never seen what the build process looks like behind the curtain on the josm server - will this work? Comments and improvements very welcome.

#!/bin/bash

set -Eeuo pipefail
# Don't leak my tokens
set +x
# Don't glob
set -f

if [ -z "${1-}" ]; then
    echo "Usage: $0 josm_revision"
    echo "For example: $0 17329"
    exit 1
else
    josm_revision="${1-}"
fi

github_repo='openstreetmap/josm'
# Grab yours for free today exclusively at https://github.com/settings/tokens
github_oauth_token='invalidtoken'

if [[ "$josm_revision" == "$(curl --silent https://josm.openstreetmap.de/tested)" ]]; then
    echo "$josm_revision is indeed the current tested release."
else
    echo "Be careful: $josm_revision is not the current tested release at https://josm.openstreetmap.de/tested."
fi

# echo "Fetching re-run url for $josm_revision from past actions in $github_repo"

if rerun_url="$(
    curl \
        --silent \
        --fail \
        --header 'Accept: application/vnd.github.v3+json' \
        --header "Authorization: token $github_oauth_token" \
        --header 'Content-Type: application/json' \
        https://api.github.com/repos/$github_repo/actions/runs |
        jq \
            --raw-output \
            --arg JOSM_REVISION "$josm_revision" \
            '.["workflow_runs"] | .[] | select(.head_commit.message | contains("\n\ngit-svn-id: https://josm.openstreetmap.de/svn/trunk@" + $JOSM_REVISION)) | .rerun_url'
)"; then
    if [ -z "$rerun_url" ]; then
        echo >&2 "Could not get rerun_url for commit matching $1 in $github_repo. Have a look at https://api.github.com/repos/$github_repo/actions/runs"
        exit 1
    else
        # echo "I got $rerun_url which I will now request"
        if test "$(
            curl \
                --silent \
                --fail \
                -X POST \
                --header 'Accept: application/vnd.github.v3+json' \
                --header "Authorization: token $github_oauth_token" \
                "$rerun_url"
        )"; then
            echo "Re-launched the build for $josm_revision"
        else
            echo >&2 -n "Could not re-launch the build for $josm_revision in $github_repo. Message: "
            # Re-fetching with curl since --fail earlier doesn't let us capture output :/
            curl --silent -X POST \
                --header 'Accept: application/vnd.github.v3+json' \
                --header "Authorization: token $github_oauth_token" \
                "$rerun_url" | jq -r .message 1>&2
            exit 1
        fi
    fi
else
    echo >&2 "Could not get runs for git-svn-id $1 in $github_repo. Did you maybe push on top of it in git  ? Have a look at https://api.github.com/repos/$github_repo/actions/runs"
    exit 1
fi

Attachments (0)

Change History (48)

comment:1 by Don-vip, 3 years ago

Owner: changed from team to Don-vip
Status: newassigned

comment:2 by simon04, 3 years ago

Do we need to reimplement the git-svn-id parsing? It's already present in the Ant task init-git-revision-xml.

comment:3 by Stereo, 3 years ago

@simon04 We don't have a git commit hash on that server at this point, unless I'm wrong. This is the best way I've found of uniquely identifying an svn revision from https://api.github.com/repos/openstreetmap/josm/actions/runs

comment:4 by stoecker, 3 years ago

I've never seen what the build process looks like behind the curtain on the josm server - will this work?

I'll rewrite it into the used script.

Steps are:

  • Run first URl
  • Extract data
  • Run second URL?

comment:5 by stoecker, 3 years ago

Owner: changed from Don-vip to stoecker
Status: assignednew

comment:6 by Stereo, 3 years ago

Yeah, the first URL returns one rerun_url per run. We find the right svn revision ID in the first URL, and call that.

I was hoping it could live in /scripts so that it can be useful to others.

comment:7 by stoecker, 3 years ago

Nah, that will be only a few lines of additional perl code in the existing script :-)

comment:8 by stoecker, 3 years ago

I also need a way to test if build worked and what's the recent MacOS version available (i.e. a feedback).

comment:9 by Stereo, 3 years ago

And I fought myself so hard to resist writing it in Python! :)

It's possible to create a github action that triggers when a new release is published. I'll see if it can work as I expect it to.

Otherwise, an apache redirect could always link to the most recent release - github has an url alias for that.

in reply to:  9 comment:10 by stoecker, 3 years ago

Replying to Stereo:

And I fought myself so hard to resist writing it in Python! :)

It's possible to create a github action that triggers when a new release is published. I'll see if it can work as I expect it to.

No. Not in the release process. Something independent which delivers the "current state" version number, however it is. So that on the wiki page it get's visible when build failed and MacOS stays on old version. Ideally a HTTP request to a page which is easy parsable.

Otherwise, an apache redirect could always link to the most recent release - github has an url alias for that.

That would be fine. How is it?

comment:11 by simon04, 3 years ago

We could also manually git tag (such as v20.11) the relevant Git commit with the version number after a release. In my local Git repository, I've tagged all 2020 releases this way (turned out to be very helpful for tracing back changes and using git bisect).

comment:12 by Stereo, 3 years ago

For example, https://github.com/grischard/josm/releases/latest/download/JOSM-macOS-java15.zip redirects to https://github.com/grischard/josm/releases/download/17329-tested/JOSM-macOS-java15.zip . The wiki or apache could, when we have a proper release on the openstreetmap/josm github, just send people to https://github.com/openstreetmap/josm/releases/latest/download/JOSM-macOS-java15.zip .

To get the mac version number for the wiki, you could parse the github release api for openstreetmap/josm. It would look similar to https://api.github.com/repos/grischard/josm/releases/latest and you could search for something like "tag_name": "17329-tested".

comment:14 by Stereo, 3 years ago

Oh, and I'd recommend not doing a release on openstreetmap/josm for 17329 because I did a couple of changes ( https://github.com/openstreetmap/josm/compare/master...grischard:master ) and it'd just be a mess.

@simon04 sorry, didn't see your comment before I posted. Those are the differences, to make releases work, after Don-vip explained to me how my understanding of josm releases was completely flawed :)

Last edited 3 years ago by Stereo (previous) (diff)

comment:15 by Stereo, 3 years ago

https://github.com/openstreetmap/josm/pull/67 shows the corresponding changes that need to be reflected on the josm side in a more stable, readable way.

comment:16 by stoecker, 3 years ago

@Vincent: Can you take care of this till the weekend. I'd like to implement the scripting stuff at the weekend.

comment:17 by Stereo, 3 years ago

@simon04 I like your idea and I'm wondering if you have an automatic way of telling that, say, 17239 is 20.11.

comment:18 by simon04, 3 years ago

I've tagged my Git repository manually. I don't think that Git tags should be applied automatically. For many Git projects, the release workflows starts automatically after a manual (and possibly signed) Git tag.

in reply to:  16 comment:19 by Don-vip, 3 years ago

Replying to stoecker:

@Vincent: Can you take care of this till the weekend. I'd like to implement the scripting stuff at the weekend.

Sure I'll take a look tonight

comment:20 by Don-vip, 3 years ago

In 17373/josm:

see #20146 - Improvements to macOS build and github actions (patch by Stereo)

From https://github.com/openstreetmap/josm/pull/67:

  • Separate the create-release github action to pass the release url to all subsequent jobs, to be able to...
  • Upload .jar artefacts from all runs, and a separate .app for all java versions for macOS
  • Better caching for ant/junit
  • Support for the release workflow from #20146
  • A few cosmetic fixes, more readable syntax, etc.

comment:21 by Stereo, 3 years ago

Resolution: fixed
Status: newclosed

comment:22 by Stereo, 3 years ago

Keywords: macos added

comment:23 by Stereo, 3 years ago

@stoeker do you maybe want to do a test release together an evening this week?

comment:24 by Stereo, 3 years ago

Resolution: fixed
Status: closedreopened

comment:25 by Stereo, 3 years ago

Status: reopenednew

in reply to:  23 comment:26 by stoecker, 3 years ago

Replying to Stereo:

@stoeker do you maybe want to do a test release together an evening this week?

ATM I feel more like reading than programming. So probably it will have to wait till next week.

comment:27 by stoecker, 3 years ago

Milestone: 20.1221.01

comment:28 by stoecker, 3 years ago

Sorry for the delays. It's not forgotten. If possible please provide a new link for the current release until I fix this ticket. I'm a bit demotivated to do much programming currently and instead am reading many books.

comment:29 by Stereo, 3 years ago

Don-vip has updated the redirect for now.

Don't let the grey weather get you down, and enjoy your reading.

Last edited 3 years ago by Stereo (previous) (diff)

comment:30 by Stereo, 3 years ago

Cc: Don-vip added

comment:31 by Stereo, 3 years ago

Ticket #20327 has been marked as a duplicate of this ticket.

comment:32 by Don-vip, 3 years ago

Milestone: 21.0121.02

comment:33 by Stereo, 3 years ago

Milestone: 21.0221.01

This has been ready to go since November. Can it be put into production this month?

comment:34 by stoecker, 3 years ago

Milestone: 21.0121.02

Milestone renamed

comment:35 by stoecker, 3 years ago

Sorry. This time I simply forgot. I had a look now. What do I do when there are more than one run. Use the newest?

Check for 17489
Found 17489 https://api.github.com/repos/openstreetmap/josm/actions/runs/553936406/rerun
Found 17489 https://api.github.com/repos/openstreetmap/josm/actions/runs/553936403/rerun
use JSON;

sub triggergithub($) {
  my ($version) = @_;
  my $cmd = "curl --silent --fail --header 'Accept: application/vnd.github.v3+json'"
  . " --header 'Authorization: token ....'"
  . " --header 'Content-Type: application/json'"
  . " https://api.github.com/repos/openstreetmap/josm/actions/runs";
  my $d = decode_json(`$cmd`);
  print "Check for $version\n";
  for my $run (@{$d->{workflow_runs}})
  {
    if($run->{head_commit}{message} =~ /git-svn-id: https:\/\/josm\.openstreetmap\.de\/svn\/trunk\@$version /)
    {
      print "Found $version $run->{rerun_url}\n";
    }
  }
}

comment:36 by Stereo, 3 years ago

Good to hear from you again :)

Now I'm really wondering why we'd have two runs for 17489 - did you maybe trigger another test run already? The runs are supposed to run once per commit, and maybe a second time if that commit gets blessed by you to a tested release.

In any case, the latest run should be a reasonable assumption. The code should be the same anyway.

in reply to:  36 comment:37 by stoecker, 3 years ago

Replying to Stereo:

Good to hear from you again :)

Now I'm really wondering why we'd have two runs for 17489 - did you maybe trigger another test run already? The runs are supposed to run once per commit, and maybe a second time if that commit gets blessed by you to a tested release.

Happens since the recent changes:

17491 17491
17489 17489
17488 17488
17487 17487
17486
17484
17483
17482
17476
17475
17474
17472
17471

comment:38 by Stereo, 3 years ago

Ahh, one of them is probably for the CodeQL thing which I added. Damn it, Guillaume.

You can distinguish them by name, where we have "name": "Java CI" or "name": "CodeQL"

Something like this? (untested)

    if($run->{head_commit}{message} =~ /git-svn-id: https:\/\/josm\.openstreetmap\.de\/svn\/trunk\@$version / && $run->{name} == "Java CI") 

in reply to:  38 comment:39 by stoecker, 3 years ago

Replying to Stereo:

... && $run->{name} == "Java CI")

Nearly. String compare needs "eq" in perl otherwise it is a numeric compare :-)

Try

perl -e 'print "true\n" if "Hallo" == "Ballo"'

And to be fair to perl later try :-)

perl -w -e 'print "true\n" if "Hallo" == "Ballo"'

comment:40 by Stereo, 3 years ago

How can I run the final perl code with my repo token to see if it correctly triggers a rebuild?

comment:41 by stoecker, 3 years ago

Finishing

#!/usr/bin/perl -w
use JSON;
use strict;

sub triggergithub($) {
  my ($version) = @_;
  eval
  {
    my $token = "...";
    my $curlbase = "curl --silent --max-time 30"
    . " --header 'Authorization: token $token'";
    my $cmd = "$curlbase --fail https://api.github.com/repos/openstreetmap/josm/actions/runs";
    my $d = decode_json(`$cmd`);
    for my $run (@{$d->{workflow_runs}})
    {
      if($run->{head_commit}{message} =~ /git-svn-id: https:\/\/josm\.openstreetmap\.de\/svn\/trunk\@$version / && $run->{name} eq "Java CI")
      {
        my $url = $run->{rerun_url};
        print "Found $version $run->{rerun_url}\n";
        die "Strange URL $url" if $url !~ /^https:\/\/api\.github\.com\/[a-z0-9\/]+$/; # prevent security risks
        my $cmd = "$curlbase -X POST $url";
print "$cmd\n";
print `$cmd`;
      }
    }
  };
  print STDERR "Error rebuilding MacOS: $@" if $@;
}

my $ver = 17489;
print "Check for $ver\n";
triggergithub($ver);

I always get

  "message": "Must have admin rights to Repository.",
  "documentation_url": "https://docs.github.com/rest/reference/actions#re-run-a-workflow"

I added "repo", but even tried all, but it does not help.

How can I run the final perl code with my repo token to see if it correctly triggers a rebuild?

Save into file, insert the token and call "perl file".

Last edited 3 years ago by Don-vip (previous) (diff)

comment:42 by Stereo, 3 years ago

Hmm, I get a strange error - it seems to choke on the json somehow:

> perl foo.pl
Check for 17489
Error rebuilding MacOS: , or } expected while parsing object/hash, at character offset 2 (before "(end of string)") at tagati.pl line 13.

But by adapting my shell script to only run on "Java CI", I can reproduce your problem by using my token on the openstreetmap/josm repo, where I don't have full access.

Do you have both 'repo' and 'workflow' checked in the token scopes?

Last edited 3 years ago by Stereo (previous) (diff)

comment:43 by stoecker, 3 years ago

I tried different setups. Once even with everything enabled. Are we maybe missing another setting in JOSM project?

I'll generate a new token. Maybe that helps.

For your error do comment the eval and closing bracket. Then you get a more descriptive error. I'd assume the JSON is somehow invalid. Maybe curl call failed?

comment:44 by Stereo, 3 years ago

You *are* admin, right? :)

in reply to:  44 comment:45 by stoecker, 3 years ago

Replying to Stereo:

You *are* admin, right? :)

Well. No :-)

Aaargh. Seems I/we never added the relevant user to the openstreetmap group.

comment:46 by stoecker, 3 years ago

So. Copied into the relevant job. Should be called with next tested. Auto-Updating the link still missing as well as some cleanup (removing old mac stuff).

comment:47 by Stereo, 3 years ago

Fingers crossed. Thank you!

For the links, #20046 can be done and closed as soon as there's a new stable build at https://github.com/openstreetmap/josm/releases/latest/download/JOSM-macOS-java15.zip . Whether we do that through a wiki search/replace or an Apache config update is up to you :).

comment:48 by Stereo, 3 years ago

Resolution: fixed
Status: newclosed

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The owner will remain stoecker.
as The resolution will be set.
The resolution will be deleted. Next status will be 'reopened'.

Add Comment


E-mail address and name can be saved in the Preferences .
 
Note: See TracTickets for help on using tickets.