Modify

Opened 3 months ago

Closed 12 days ago

Last modified 9 days ago

#20146 closed enhancement (fixed)

Trigger macOS re-build on github when changeset marked at tested

Reported by: Stereo Owned by: stoecker
Priority: normal Milestone: 21.02
Component: Installer MacOS Version:
Keywords: macos Cc: Don-vip

Description

This is a work in progress, intended to run on the josm server when releasing a new 'tested' version. Instead of triggering a new macOS build locally like in the past, this finds the github action workflow that contained git-svn-id: https://josm.openstreetmap.de/svn/trunk@$josm_revision in its commit message.

When a git action run is triggered, it fetches https://josm.openstreetmap.de/tested. If /tested has been properly updated before triggering the rebuild, it will build as a release instead of a pre-release, like https://github.com/grischard/josm/releases/tag/17329-tested

The shell script requires jq to parse the github json.

I've never seen what the build process looks like behind the curtain on the josm server - will this work? Comments and improvements very welcome.

#!/bin/bash

set -Eeuo pipefail
# Don't leak my tokens
set +x
# Don't glob
set -f

if [ -z "${1-}" ]; then
    echo "Usage: $0 josm_revision"
    echo "For example: $0 17329"
    exit 1
else
    josm_revision="${1-}"
fi

github_repo='openstreetmap/josm'
# Grab yours for free today exclusively at https://github.com/settings/tokens
github_oauth_token='invalidtoken'

if [[ "$josm_revision" == "$(curl --silent https://josm.openstreetmap.de/tested)" ]]; then
    echo "$josm_revision is indeed the current tested release."
else
    echo "Be careful: $josm_revision is not the current tested release at https://josm.openstreetmap.de/tested."
fi

# echo "Fetching re-run url for $josm_revision from past actions in $github_repo"

if rerun_url="$(
    curl \
        --silent \
        --fail \
        --header 'Accept: application/vnd.github.v3+json' \
        --header "Authorization: token $github_oauth_token" \
        --header 'Content-Type: application/json' \
        https://api.github.com/repos/$github_repo/actions/runs |
        jq \
            --raw-output \
            --arg JOSM_REVISION "$josm_revision" \
            '.["workflow_runs"] | .[] | select(.head_commit.message | contains("\n\ngit-svn-id: https://josm.openstreetmap.de/svn/trunk@" + $JOSM_REVISION)) | .rerun_url'
)"; then
    if [ -z "$rerun_url" ]; then
        echo >&2 "Could not get rerun_url for commit matching $1 in $github_repo. Have a look at https://api.github.com/repos/$github_repo/actions/runs"
        exit 1
    else
        # echo "I got $rerun_url which I will now request"
        if test "$(
            curl \
                --silent \
                --fail \
                -X POST \
                --header 'Accept: application/vnd.github.v3+json' \
                --header "Authorization: token $github_oauth_token" \
                "$rerun_url"
        )"; then
            echo "Re-launched the build for $josm_revision"
        else
            echo >&2 -n "Could not re-launch the build for $josm_revision in $github_repo. Message: "
            # Re-fetching with curl since --fail earlier doesn't let us capture output :/
            curl --silent -X POST \
                --header 'Accept: application/vnd.github.v3+json' \
                --header "Authorization: token $github_oauth_token" \
                "$rerun_url" | jq -r .message 1>&2
            exit 1
        fi
    fi
else
    echo >&2 "Could not get runs for git-svn-id $1 in $github_repo. Did you maybe push on top of it in git  ? Have a look at https://api.github.com/repos/$github_repo/actions/runs"
    exit 1
fi

Attachments (0)

Change History (48)

comment:1 Changed 3 months ago by Don-vip

Owner: changed from team to Don-vip
Status: newassigned

comment:2 Changed 3 months ago by simon04

Do we need to reimplement the git-svn-id parsing? It's already present in the Ant task init-git-revision-xml.

comment:3 Changed 3 months ago by Stereo

@simon04 We don't have a git commit hash on that server at this point, unless I'm wrong. This is the best way I've found of uniquely identifying an svn revision from https://api.github.com/repos/openstreetmap/josm/actions/runs

comment:4 Changed 3 months ago by stoecker

I've never seen what the build process looks like behind the curtain on the josm server - will this work?

I'll rewrite it into the used script.

Steps are:

  • Run first URl
  • Extract data
  • Run second URL?

comment:5 Changed 3 months ago by stoecker

Owner: changed from Don-vip to stoecker
Status: assignednew

comment:6 Changed 3 months ago by Stereo

Yeah, the first URL returns one rerun_url per run. We find the right svn revision ID in the first URL, and call that.

I was hoping it could live in /scripts so that it can be useful to others.

comment:7 Changed 3 months ago by stoecker

Nah, that will be only a few lines of additional perl code in the existing script :-)

comment:8 Changed 3 months ago by stoecker

I also need a way to test if build worked and what's the recent MacOS version available (i.e. a feedback).

comment:9 Changed 3 months ago by Stereo

And I fought myself so hard to resist writing it in Python! :)

It's possible to create a github action that triggers when a new release is published. I'll see if it can work as I expect it to.

Otherwise, an apache redirect could always link to the most recent release - github has an url alias for that.

comment:10 in reply to:  9 Changed 3 months ago by stoecker

Replying to Stereo:

And I fought myself so hard to resist writing it in Python! :)

It's possible to create a github action that triggers when a new release is published. I'll see if it can work as I expect it to.

No. Not in the release process. Something independent which delivers the "current state" version number, however it is. So that on the wiki page it get's visible when build failed and MacOS stays on old version. Ideally a HTTP request to a page which is easy parsable.

Otherwise, an apache redirect could always link to the most recent release - github has an url alias for that.

That would be fine. How is it?

comment:11 Changed 3 months ago by simon04

We could also manually git tag (such as v20.11) the relevant Git commit with the version number after a release. In my local Git repository, I've tagged all 2020 releases this way (turned out to be very helpful for tracing back changes and using git bisect).

comment:12 Changed 3 months ago by Stereo

For example, https://github.com/grischard/josm/releases/latest/download/JOSM-macOS-java15.zip redirects to https://github.com/grischard/josm/releases/download/17329-tested/JOSM-macOS-java15.zip . The wiki or apache could, when we have a proper release on the openstreetmap/josm github, just send people to https://github.com/openstreetmap/josm/releases/latest/download/JOSM-macOS-java15.zip .

To get the mac version number for the wiki, you could parse the github release api for openstreetmap/josm. It would look similar to https://api.github.com/repos/grischard/josm/releases/latest and you could search for something like "tag_name": "17329-tested".

comment:14 Changed 3 months ago by Stereo

Oh, and I'd recommend not doing a release on openstreetmap/josm for 17329 because I did a couple of changes ( https://github.com/openstreetmap/josm/compare/master...grischard:master ) and it'd just be a mess.

@simon04 sorry, didn't see your comment before I posted. Those are the differences, to make releases work, after Don-vip explained to me how my understanding of josm releases was completely flawed :)

Last edited 3 months ago by Stereo (previous) (diff)

comment:15 Changed 3 months ago by Stereo

https://github.com/openstreetmap/josm/pull/67 shows the corresponding changes that need to be reflected on the josm side in a more stable, readable way.

comment:16 Changed 3 months ago by stoecker

@Vincent: Can you take care of this till the weekend. I'd like to implement the scripting stuff at the weekend.

comment:17 Changed 3 months ago by Stereo

@simon04 I like your idea and I'm wondering if you have an automatic way of telling that, say, 17239 is 20.11.

comment:18 Changed 3 months ago by simon04

I've tagged my Git repository manually. I don't think that Git tags should be applied automatically. For many Git projects, the release workflows starts automatically after a manual (and possibly signed) Git tag.

comment:19 in reply to:  16 Changed 3 months ago by Don-vip

Replying to stoecker:

@Vincent: Can you take care of this till the weekend. I'd like to implement the scripting stuff at the weekend.

Sure I'll take a look tonight

comment:20 Changed 3 months ago by Don-vip

In 17373/josm:

see #20146 - Improvements to macOS build and github actions (patch by Stereo)

From https://github.com/openstreetmap/josm/pull/67:

  • Separate the create-release github action to pass the release url to all subsequent jobs, to be able to...
  • Upload .jar artefacts from all runs, and a separate .app for all java versions for macOS
  • Better caching for ant/junit
  • Support for the release workflow from #20146
  • A few cosmetic fixes, more readable syntax, etc.

comment:21 Changed 3 months ago by Stereo

Resolution: fixed
Status: newclosed

comment:22 Changed 3 months ago by Stereo

Keywords: macos added

comment:23 Changed 3 months ago by Stereo

@stoeker do you maybe want to do a test release together an evening this week?

comment:24 Changed 3 months ago by Stereo

Resolution: fixed
Status: closedreopened

comment:25 Changed 3 months ago by Stereo

Status: reopenednew

comment:26 in reply to:  23 Changed 3 months ago by stoecker

Replying to Stereo:

@stoeker do you maybe want to do a test release together an evening this week?

ATM I feel more like reading than programming. So probably it will have to wait till next week.

comment:27 Changed 2 months ago by stoecker

Milestone: 20.1221.01

comment:28 Changed 2 months ago by stoecker

Sorry for the delays. It's not forgotten. If possible please provide a new link for the current release until I fix this ticket. I'm a bit demotivated to do much programming currently and instead am reading many books.

comment:29 Changed 2 months ago by Stereo

Please update the redirect

https://josm.openstreetmap.de/download/macosx/josm-macosx.zip

to go to

https://github.com/grischard/josm/releases/download/17428-tested/JOSM-macOS-java15.zip

for now.

Don't let the grey weather get you down, and enjoy your reading.

Version 0, edited 2 months ago by Stereo (next)

comment:30 Changed 2 months ago by Stereo

Cc: Don-vip added

comment:31 Changed 2 months ago by Stereo

Ticket #20327 has been marked as a duplicate of this ticket.

comment:32 Changed 4 weeks ago by Don-vip

Milestone: 21.0121.02

comment:33 Changed 3 weeks ago by Stereo

Milestone: 21.0221.01

This has been ready to go since November. Can it be put into production this month?

comment:34 Changed 3 weeks ago by stoecker

Milestone: 21.0121.02

Milestone renamed

comment:35 Changed 3 weeks ago by stoecker

Sorry. This time I simply forgot. I had a look now. What do I do when there are more than one run. Use the newest?

Check for 17489
Found 17489 https://api.github.com/repos/openstreetmap/josm/actions/runs/553936406/rerun
Found 17489 https://api.github.com/repos/openstreetmap/josm/actions/runs/553936403/rerun
use JSON;

sub triggergithub($) {
  my ($version) = @_;
  my $cmd = "curl --silent --fail --header 'Accept: application/vnd.github.v3+json'"
  . " --header 'Authorization: token ....'"
  . " --header 'Content-Type: application/json'"
  . " https://api.github.com/repos/openstreetmap/josm/actions/runs";
  my $d = decode_json(`$cmd`);
  print "Check for $version\n";
  for my $run (@{$d->{workflow_runs}})
  {
    if($run->{head_commit}{message} =~ /git-svn-id: https:\/\/josm\.openstreetmap\.de\/svn\/trunk\@$version /)
    {
      print "Found $version $run->{rerun_url}\n";
    }
  }
}

comment:36 Changed 2 weeks ago by Stereo

Good to hear from you again :)

Now I'm really wondering why we'd have two runs for 17489 - did you maybe trigger another test run already? The runs are supposed to run once per commit, and maybe a second time if that commit gets blessed by you to a tested release.

In any case, the latest run should be a reasonable assumption. The code should be the same anyway.

comment:37 in reply to:  36 Changed 2 weeks ago by stoecker

Replying to Stereo:

Good to hear from you again :)

Now I'm really wondering why we'd have two runs for 17489 - did you maybe trigger another test run already? The runs are supposed to run once per commit, and maybe a second time if that commit gets blessed by you to a tested release.

Happens since the recent changes:

17491 17491
17489 17489
17488 17488
17487 17487
17486
17484
17483
17482
17476
17475
17474
17472
17471

comment:38 Changed 2 weeks ago by Stereo

Ahh, one of them is probably for the CodeQL thing which I added. Damn it, Guillaume.

You can distinguish them by name, where we have "name": "Java CI" or "name": "CodeQL"

Something like this? (untested)

    if($run->{head_commit}{message} =~ /git-svn-id: https:\/\/josm\.openstreetmap\.de\/svn\/trunk\@$version / && $run->{name} == "Java CI") 

comment:39 in reply to:  38 Changed 2 weeks ago by stoecker

Replying to Stereo:

... && $run->{name} == "Java CI")

Nearly. String compare needs "eq" in perl otherwise it is a numeric compare :-)

Try

perl -e 'print "true\n" if "Hallo" == "Ballo"'

And to be fair to perl later try :-)

perl -w -e 'print "true\n" if "Hallo" == "Ballo"'

comment:40 Changed 2 weeks ago by Stereo

How can I run the final perl code with my repo token to see if it correctly triggers a rebuild?

comment:41 Changed 12 days ago by stoecker

Finishing

#!/usr/bin/perl -w
use JSON;
use strict;

sub triggergithub($) {
  my ($version) = @_;
  eval
  {
    my $token = "...";
    my $curlbase = "curl --silent --max-time 30"
    . " --header 'Authorization: token $token'";
    my $cmd = "$curlbase --fail https://api.github.com/repos/openstreetmap/josm/actions/runs";
    my $d = decode_json(`$cmd`);
    for my $run (@{$d->{workflow_runs}})
    {
      if($run->{head_commit}{message} =~ /git-svn-id: https:\/\/josm\.openstreetmap\.de\/svn\/trunk\@$version / && $run->{name} eq "Java CI")
      {
        my $url = $run->{rerun_url};
        print "Found $version $run->{rerun_url}\n";
        die "Strange URL $url" if $url !~ /^https:\/\/api\.github\.com\/[a-z0-9\/]+$/; # prevent security risks
        my $cmd = "$curlbase -X POST $url";
print "$cmd\n";
print `$cmd`;
      }
    }
  };
  print STDERR "Error rebuilding MacOS: $@" if $@;
}

my $ver = 17489;
print "Check for $ver\n";
triggergithub($ver);

I always get

  "message": "Must have admin rights to Repository.",
  "documentation_url": "https://docs.github.com/rest/reference/actions#re-run-a-workflow"

I added "repo", but even tried all, but it does not help.

How can I run the final perl code with my repo token to see if it correctly triggers a rebuild?

Save into file, insert the token and call "perl file".

Last edited 9 days ago by Don-vip (previous) (diff)

comment:42 Changed 12 days ago by Stereo

Hmm, I get a strange error - it seems to choke on the json somehow:

> perl foo.pl
Check for 17489
Error rebuilding MacOS: , or } expected while parsing object/hash, at character offset 2 (before "(end of string)") at tagati.pl line 13.

But by adapting my shell script to only run on "Java CI", I can reproduce your problem by using my token on the openstreetmap/josm repo, where I don't have full access.

Do you have both 'repo' and 'workflow' checked in the token scopes?

Last edited 12 days ago by Stereo (previous) (diff)

comment:43 Changed 12 days ago by stoecker

I tried different setups. Once even with everything enabled. Are we maybe missing another setting in JOSM project?

I'll generate a new token. Maybe that helps.

For your error do comment the eval and closing bracket. Then you get a more descriptive error. I'd assume the JSON is somehow invalid. Maybe curl call failed?

comment:44 Changed 12 days ago by Stereo

You *are* admin, right? :)

comment:45 in reply to:  44 Changed 12 days ago by stoecker

Replying to Stereo:

You *are* admin, right? :)

Well. No :-)

Aaargh. Seems I/we never added the relevant user to the openstreetmap group.

comment:46 Changed 12 days ago by stoecker

So. Copied into the relevant job. Should be called with next tested. Auto-Updating the link still missing as well as some cleanup (removing old mac stuff).

comment:47 Changed 12 days ago by Stereo

Fingers crossed. Thank you!

For the links, #20046 can be done and closed as soon as there's a new stable build at https://github.com/openstreetmap/josm/releases/latest/download/JOSM-macOS-java15.zip . Whether we do that through a wiki search/replace or an Apache config update is up to you :).

comment:48 Changed 12 days ago by Stereo

Resolution: fixed
Status: newclosed

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The owner will remain stoecker.
as The resolution will be set.
The resolution will be deleted.

Add Comment


E-mail address and name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.