Modify

Opened 2 months ago

Last modified 3 weeks ago

#20146 new enhancement

Trigger macOS re-build on github when changeset marked at tested

Reported by: Stereo Owned by: stoecker
Priority: normal Milestone: 21.01
Component: Installer MacOS Version:
Keywords: macos Cc: Don-vip

Description

This is a work in progress, intended to run on the josm server when releasing a new 'tested' version. Instead of triggering a new macOS build locally like in the past, this finds the github action workflow that contained git-svn-id: https://josm.openstreetmap.de/svn/trunk@$josm_revision in its commit message.

When a git action run is triggered, it fetches https://josm.openstreetmap.de/tested. If /tested has been properly updated before triggering the rebuild, it will build as a release instead of a pre-release, like https://github.com/grischard/josm/releases/tag/17329-tested

The shell script requires jq to parse the github json.

I've never seen what the build process looks like behind the curtain on the josm server - will this work? Comments and improvements very welcome.

#!/bin/bash

set -Eeuo pipefail
# Don't leak my tokens
set +x
# Don't glob
set -f

if [ -z "${1-}" ]; then
    echo "Usage: $0 josm_revision"
    echo "For example: $0 17329"
    exit 1
else
    josm_revision="${1-}"
fi

github_repo='openstreetmap/josm'
# Grab yours for free today exclusively at https://github.com/settings/tokens
github_oauth_token='invalidtoken'

if [[ "$josm_revision" == "$(curl --silent https://josm.openstreetmap.de/tested)" ]]; then
    echo "$josm_revision is indeed the current tested release."
else
    echo "Be careful: $josm_revision is not the current tested release at https://josm.openstreetmap.de/tested."
fi

# echo "Fetching re-run url for $josm_revision from past actions in $github_repo"

if rerun_url="$(
    curl \
        --silent \
        --fail \
        --header 'Accept: application/vnd.github.v3+json' \
        --header "Authorization: token $github_oauth_token" \
        --header 'Content-Type: application/json' \
        https://api.github.com/repos/$github_repo/actions/runs |
        jq \
            --raw-output \
            --arg JOSM_REVISION "$josm_revision" \
            '.["workflow_runs"] | .[] | select(.head_commit.message | contains("\n\ngit-svn-id: https://josm.openstreetmap.de/svn/trunk@" + $JOSM_REVISION)) | .rerun_url'
)"; then
    if [ -z "$rerun_url" ]; then
        echo >&2 "Could not get rerun_url for commit matching $1 in $github_repo. Have a look at https://api.github.com/repos/$github_repo/actions/runs"
        exit 1
    else
        # echo "I got $rerun_url which I will now request"
        if test "$(
            curl \
                --silent \
                --fail \
                -X POST \
                --header 'Accept: application/vnd.github.v3+json' \
                --header "Authorization: token $github_oauth_token" \
                "$rerun_url"
        )"; then
            echo "Re-launched the build for $josm_revision"
        else
            echo >&2 -n "Could not re-launch the build for $josm_revision in $github_repo. Message: "
            # Re-fetching with curl since --fail earlier doesn't let us capture output :/
            curl --silent -X POST \
                --header 'Accept: application/vnd.github.v3+json' \
                --header "Authorization: token $github_oauth_token" \
                "$rerun_url" | jq -r .message 1>&2
            exit 1
        fi
    fi
else
    echo >&2 "Could not get runs for git-svn-id $1 in $github_repo. Did you maybe push on top of it in git  ? Have a look at https://api.github.com/repos/$github_repo/actions/runs"
    exit 1
fi

Attachments (0)

Change History (31)

comment:1 Changed 2 months ago by Don-vip

Owner: changed from team to Don-vip
Status: newassigned

comment:2 Changed 2 months ago by simon04

Do we need to reimplement the git-svn-id parsing? It's already present in the Ant task init-git-revision-xml.

comment:3 Changed 2 months ago by Stereo

@simon04 We don't have a git commit hash on that server at this point, unless I'm wrong. This is the best way I've found of uniquely identifying an svn revision from https://api.github.com/repos/openstreetmap/josm/actions/runs

comment:4 Changed 2 months ago by stoecker

I've never seen what the build process looks like behind the curtain on the josm server - will this work?

I'll rewrite it into the used script.

Steps are:

  • Run first URl
  • Extract data
  • Run second URL?

comment:5 Changed 2 months ago by stoecker

Owner: changed from Don-vip to stoecker
Status: assignednew

comment:6 Changed 2 months ago by Stereo

Yeah, the first URL returns one rerun_url per run. We find the right svn revision ID in the first URL, and call that.

I was hoping it could live in /scripts so that it can be useful to others.

comment:7 Changed 2 months ago by stoecker

Nah, that will be only a few lines of additional perl code in the existing script :-)

comment:8 Changed 2 months ago by stoecker

I also need a way to test if build worked and what's the recent MacOS version available (i.e. a feedback).

comment:9 Changed 2 months ago by Stereo

And I fought myself so hard to resist writing it in Python! :)

It's possible to create a github action that triggers when a new release is published. I'll see if it can work as I expect it to.

Otherwise, an apache redirect could always link to the most recent release - github has an url alias for that.

comment:10 in reply to:  9 Changed 2 months ago by stoecker

Replying to Stereo:

And I fought myself so hard to resist writing it in Python! :)

It's possible to create a github action that triggers when a new release is published. I'll see if it can work as I expect it to.

No. Not in the release process. Something independent which delivers the "current state" version number, however it is. So that on the wiki page it get's visible when build failed and MacOS stays on old version. Ideally a HTTP request to a page which is easy parsable.

Otherwise, an apache redirect could always link to the most recent release - github has an url alias for that.

That would be fine. How is it?

comment:11 Changed 2 months ago by simon04

We could also manually git tag (such as v20.11) the relevant Git commit with the version number after a release. In my local Git repository, I've tagged all 2020 releases this way (turned out to be very helpful for tracing back changes and using git bisect).

comment:12 Changed 2 months ago by Stereo

For example, https://github.com/grischard/josm/releases/latest/download/JOSM-macOS-java15.zip redirects to https://github.com/grischard/josm/releases/download/17329-tested/JOSM-macOS-java15.zip . The wiki or apache could, when we have a proper release on the openstreetmap/josm github, just send people to https://github.com/openstreetmap/josm/releases/latest/download/JOSM-macOS-java15.zip .

To get the mac version number for the wiki, you could parse the github release api for openstreetmap/josm. It would look similar to https://api.github.com/repos/grischard/josm/releases/latest and you could search for something like "tag_name": "17329-tested".

comment:14 Changed 2 months ago by Stereo

Oh, and I'd recommend not doing a release on openstreetmap/josm for 17329 because I did a couple of changes ( https://github.com/openstreetmap/josm/compare/master...grischard:master ) and it'd just be a mess.

@simon04 sorry, didn't see your comment before I posted. Those are the differences, to make releases work, after Don-vip explained to me how my understanding of josm releases was completely flawed :)

Last edited 2 months ago by Stereo (previous) (diff)

comment:15 Changed 8 weeks ago by Stereo

https://github.com/openstreetmap/josm/pull/67 shows the corresponding changes that need to be reflected on the josm side in a more stable, readable way.

comment:16 Changed 8 weeks ago by stoecker

@Vincent: Can you take care of this till the weekend. I'd like to implement the scripting stuff at the weekend.

comment:17 Changed 8 weeks ago by Stereo

@simon04 I like your idea and I'm wondering if you have an automatic way of telling that, say, 17239 is 20.11.

comment:18 Changed 8 weeks ago by simon04

I've tagged my Git repository manually. I don't think that Git tags should be applied automatically. For many Git projects, the release workflows starts automatically after a manual (and possibly signed) Git tag.

comment:19 in reply to:  16 Changed 8 weeks ago by Don-vip

Replying to stoecker:

@Vincent: Can you take care of this till the weekend. I'd like to implement the scripting stuff at the weekend.

Sure I'll take a look tonight

comment:20 Changed 8 weeks ago by Don-vip

In 17373/josm:

see #20146 - Improvements to macOS build and github actions (patch by Stereo)

From https://github.com/openstreetmap/josm/pull/67:

  • Separate the create-release github action to pass the release url to all subsequent jobs, to be able to...
  • Upload .jar artefacts from all runs, and a separate .app for all java versions for macOS
  • Better caching for ant/junit
  • Support for the release workflow from #20146
  • A few cosmetic fixes, more readable syntax, etc.

comment:21 Changed 8 weeks ago by Stereo

Resolution: fixed
Status: newclosed

comment:22 Changed 8 weeks ago by Stereo

Keywords: macos added

comment:23 Changed 8 weeks ago by Stereo

@stoeker do you maybe want to do a test release together an evening this week?

comment:24 Changed 8 weeks ago by Stereo

Resolution: fixed
Status: closedreopened

comment:25 Changed 8 weeks ago by Stereo

Status: reopenednew

comment:26 in reply to:  23 Changed 8 weeks ago by stoecker

Replying to Stereo:

@stoeker do you maybe want to do a test release together an evening this week?

ATM I feel more like reading than programming. So probably it will have to wait till next week.

comment:27 Changed 3 weeks ago by stoecker

Milestone: 20.1221.01

comment:28 Changed 3 weeks ago by stoecker

Sorry for the delays. It's not forgotten. If possible please provide a new link for the current release until I fix this ticket. I'm a bit demotivated to do much programming currently and instead am reading many books.

comment:29 Changed 3 weeks ago by Stereo

Don-vip has updated the redirect for now.

Don't let the grey weather get you down, and enjoy your reading.

Last edited 3 weeks ago by Stereo (previous) (diff)

comment:30 Changed 3 weeks ago by Stereo

Cc: Don-vip added

comment:31 Changed 3 weeks ago by Stereo

Ticket #20327 has been marked as a duplicate of this ticket.

Modify Ticket

Change Properties
Set your email in Preferences
Action
as new The owner will remain stoecker.
as The resolution will be set.
to The owner will be changed from stoecker to the specified user.
The owner will change to Stereo
as duplicate The resolution will be set to duplicate.The specified ticket will be cross-referenced with this ticket
The owner will be changed from stoecker to anonymous.

Add Comment


E-mail address and name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.