Quick-example pyproject.toml
An example pyprojet.toml
configuration file, used for the demo code in the
desc-continuous-integraion
repository.
1[build-system]
2requires = ["setuptools >= 61.0"] # PEP 621 compliant
3build-backend = "setuptools.build_meta"
4
5[project]
6name = "mydescpackage"
7description = "Example DESC Python package, some simple mathmatical functions."
8readme = "README.md"
9authors = [{ name = "Stuart McAlpine", email = "stuart.mcalpine@fysik.su.se" }]
10license = { file = "LICENCE" }
11classifiers = [
12 "Programming Language :: Python :: 3",
13]
14keywords = ["desc", "python"]
15dependencies = [
16 'numpy',
17]
18requires-python = ">=3.8"
19dynamic = ["version"] # Scrape the version dynamically from the package
20
21[tool.setuptools.dynamic]
22version = {attr = "mydescpackage._version.__version__"}
23
24[tool.setuptools.packages.find]
25where = ["src"]
26
27[project.optional-dependencies]
28ci = ["pytest", "pytest-cov", "flake8"]
29
30[project.scripts]
31display-pi = "mydescpackage.pi:display_pi"
Quick-example ci_workflow.yml
An example Continuous Integration YAML workflow
(.github/workflows/ci_example_2.yml
in the desc-continuous-integraion
repository). More details in GitHub Actions.
1# Author - Stuart McAlpine - stuart.mcalpine@fysik.su.se - Jan 2023
2
3name: DESC Example 2
4
5# How does the workflow get triggered?
6on:
7 # Triggers when push/pull-request made to the main branch.
8 pull_request:
9 branches:
10 - main
11 push:
12 branches:
13 - main
14
15# List of jobs for this workflow.
16jobs:
17
18 # Our pytest job.
19 ci-with-pytest:
20
21 # Our strategy lists the OS and Python versions we want to test on.
22 strategy:
23
24 # Don't quit all jobs if only one job fails.
25 fail-fast: false
26
27 matrix:
28 python-version: ["3.8","3.9","3.10","3.11","3.12"]
29 os: [ubuntu-20.04, ubuntu-latest, macos-latest]
30 experimental: [false]
31
32 # Test on this, but don't mind if it fails.
33 include:
34 - os: ubuntu-latest
35 python-version: "3.11"
36 experimental: true
37
38 # If True, do not fail the job, just warn me.
39 continue-on-error: ${{ matrix.experimental }}
40
41 # What operating system is this job running on?
42 runs-on: ${{ matrix.os }}
43
44 # Our CI steps for this job.
45 steps:
46 # Check out this repository code.
47 - name: Check out repository code
48 uses: actions/checkout@v3
49
50 # Install Python.
51 - name: Set up Python ${{ matrix.python-version }}
52 uses: actions/setup-python@v4
53 with:
54 python-version: ${{ matrix.python-version }}
55
56 # Install my package.
57 - name: Install mydescpackage.
58 run: |
59 python -m pip install --upgrade pip
60 python -m pip install .[ci]
61
62 # Do some basic code linting using flake8. Check for syntax and indentation errors.
63 - name: flake8 linting.
64 run: flake8 --count --select=E1,E9 --show-source --statistics ./src/mydescpackage/*.py
65
66 # Perform the unit tests and output a coverage report.
67 - name: Test with pytest
68 run: pytest --cov=mydescpackage --cov-report xml ./tests
69
70 # Upload the code coverage results to codecov.io.
71 - name: Upload coverage to Codecov
72 uses: codecov/codecov-action@v2
NERSC CI files
Below are some details of the workings of the mirror
and target
repositories’ CI workflows.
Mirror repository files
The purpose of the mirror
repository is to clone the contents of the
source
repository at GitHub to the target
repository at the NERSC
GitLab instance. This is done through a simple CI workflow (.gitlab-ci.yml
)
which runs a simple Bash script (mirror.bash
), and is automatically
triggered from the CI workflow at the GitHub source
repository.
The two files that make up the CI workflow for the mirror
repository are
detailed below.
1# ----------------------------------------------------------------------------#
2# Script to clone a DESC repository from GitHub to the NERSC GitLab instance. #
3# #
4# This script goes in your "mirror" repository on GitLab. #
5# #
6# Script adapted from the CI Resources "mirroring" example #
7# (https://software.nersc.gov/ci-resources/mirroring) #
8# #
9# You need to set these in Settings->CI/CD->Variables: #
10# - MIRROR_SOURCE_PAT : PAT of the source repository from GitHub. #
11# - MIRROR_TARGET_PAT : PAT of the target repository on GitLab. #
12# #
13# Other variables are passed along with the trigger token from GitHub. #
14# #
15# Authors: #
16# Stuart McAlpine (@stuartmcalpine) #
17# Heather Kelly (@heather999) #
18# ----------------------------------------------------------------------------#
19
20#!/bin/bash
21
22set -e
23
24# Working directory.
25wd=$(pwd)
26
27# Always want to remove authentication file on exit (file even if workflow fails).
28function cleanup()
29{
30 echo "Cleanup: Removing ${wd}/askpass"
31 rm -rf ${wd}/askpass
32}
33trap cleanup EXIT
34
35# Set GIT_ASKPASS for git authentication.
36function set_askpass()
37{
38 echo "#!/bin/bash" > ${wd}/askpass
39 echo "echo $1" >> ${wd}/askpass
40
41 chmod +x ${wd}/askpass
42 export GIT_ASKPASS=${wd}/askpass
43}
44
45# Step 1) Fetch information from source repository.
46set_askpass ${MIRROR_SOURCE_PAT}
47echo "Cloning from ${GIBHUB_SOURCE_REPO}"
48
49rm -rf source
50git init --bare source
51
52cd source
53git remote add origin ${GITHUB_SOURCE_REPO}
54git config remote.origin.mirror true
55
56git fetch origin --prune
57
58# Step 2) Clone source repository contents to target repository.
59set_askpass ${MIRROR_TARGET_PAT}
60git remote add target ${GITLAB_TARGET_REPO}
61
62# Only pushes the MIRROR_SOURCE_BRANCH repository (and its tags).
63# Protected branches do not translate across gitlab, github, bitbucket etc.
64git push target --prune +refs/remotes/origin/$GITHUB_SOURCE_BRANCH:refs/heads/$GITHUB_SOURCE_BRANCH +refs/tags/*:refs/tags/*
The git push target --prune
means we are only pushing the
$GITHUB_SOURCE_BRANCH
branch (and tags) of the source
repository (which
will likely be your main
or master
branch). It is possible to target
multiple branches through this mechanism, however it not recommended to mirror
all branches by default.
Additionally, we also use the environment variable GIT_ASKPASS
to provide
authentication tokens to git. This ensures that the tokens are not visible in
the CLI which is an important requirement in multi-tenant hosts e.g. Cori
nodes.
1variables:
2 # Cori queue submission options.
3 SCHEDULER_PARAMETERS: "-C haswell -q debug -N1 -t 00:05:00"
4
5# How is the workflow triggered?
6workflow:
7 rules:
8 # Can only be initiated via a trigger token.
9 - if: $CI_PIPELINE_SOURCE == "trigger"
10 when: always
11
12# The CI Job.
13mirror-repo:
14
15 # Running on Cori.
16 tags: [cori]
17
18 script:
19 # Script to mirror the GitHub repository.
20 - bash mirror.bash
21
22 # Trigger the GitLab CI workflow in the target repository on GitLab.
23 - >
24 curl -X POST --fail
25 --form "variables[GITLAB_STATUS_REPO]=$GITLAB_STATUS_REPO"
26 --form "variables[GITLAB_TARGET_REPO]=$GITLAB_TARGET_REPO"
27 --form "variables[GITLAB_MIRROR_REPO]=$GITLAB_MIRROR_REPO"
28 --form "variables[GITHUB_SOURCE_REPO]=$GITHUB_SOURCE_REPO"
29 --form "variables[GITHUB_SOURCE_BRANCH]=$GITHUB_SOURCE_BRANCH"
30 --form "variables[GITLAB_TARGET_PROJECT_NUMBER]=$GITLAB_TARGET_PROJECT_NUMBER"
31 --form "variables[GITLAB_MIRROR_PROJECT_NUMBER]=$GITLAB_MIRROR_PROJECT_NUMBER"
32 --form "variables[GITLAB_STATUS_PROJECT_NUMBER]=$GITLAB_STATUS_PROJECT_NUMBER"
33 --form "variables[GITLAB_STATUS_CONTEXT]=$GITLAB_STATUS_CONTEXT"
34 --form "variables[GITHUB_SHA]=$GITHUB_SHA"
35 --form token=$TARGET_TRIGGER_TOKEN
36 --form ref=$GITHUB_SOURCE_BRANCH
37 https://software.nersc.gov/api/v4/projects/$GITLAB_TARGET_PROJECT_NUMBER/trigger/pipeline
Similar to the workflows for GitHub Actions, GitLab workflows are defined in a
file named .gitlab-ci.yml
, which must reside in the root directory of your
repository.
This workflow…
can only be triggered via a trigger token (which comes from the GitHub CI job).
runs the
mirror.bash
script, which clones thesource
repository into thetarget
repository.automatically triggers the CI workflow of the
target
repository, which continues the pipeline.
When submitting a CI workflow to Cori you must include the queue submission
parameters, passed as the environment variable SCHEDULER_PARAMETERS
. The
format is the same as if you were submitting a standard job at NERSC. Our job
(cloning a repository) is very quick, so we are submitting to the debug
queue. However if you need a longer job runtime, or want to charge the time to
a particular project, you will need to modify these parameters accordingly.
Note
The repo URLs (GITLAB_TARGET_REPO
etc) and project numbers
(GITLAB_TARGET_PROJECT_NUMBER
etc) have been passed through along with
the trigger token, originally defined in the source
CI workflow.
Status repository files
The status-github.py
file, shown below, is a Python script that
automatically appends the CI workflow result from the target
repository on
GitLab to the source
repository on GitHub.
1# ----------------------------------------------------------------------------#
2# Script to report the status of the CI workflow of the target repository on #
3# GitLab to the source repository on GitHub. #
4# #
5# This script goes in your "status" repository on GitLab. #
6# #
7# Script adapted from the CI Resources "Report Status" example #
8# (https://software.nersc.gov/ci-resources/report-status) #
9# #
10# You need to set these in Settings->CI/CD->Variables: #
11# - STATUS_SOURCE_PAT : PAT of the target repository from GitLab. #
12# - STATUS_TARGET_PAT : PAT of the source repository on GitHub. #
13# #
14# Other variables are passed along with the trigger token from GitHub. #
15# #
16# Authors: #
17# Stuart McAlpine (@stuartmcalpine) #
18# Heather Kelly (@heather999) #
19# ----------------------------------------------------------------------------#
20
21#!/usr/bin/env python3
22import os
23import requests
24import json
25import time
26
27# Pull out environment variables.
28source_repo = os.getenv("GITLAB_TARGET_REPO")
29source_branch = os.getenv("GITHUB_SOURCE_BRANCH")
30target_repo = os.getenv("GITHUB_SOURCE_REPO")
31source_pat = os.getenv("STATUS_SOURCE_PAT")
32target_pat = os.getenv("STATUS_TARGET_PAT")
33target_repo_context = os.getenv("GITLAB_STATUS_CONTEXT")
34sha = os.getenv("GITHUB_SHA")
35
36# Get information from repos APIs.
37source_api = "https://software.nersc.gov/api/v4"
38target_api = "https://api.github.com"
39
40target_project = target_repo.split("/")[-2:]
41seperator = "/"
42target_project = seperator.join(target_project)
43target_project = target_project.split(".")[0]
44
45source_project = source_repo.split("/")[-1].split(".")[0]
46search_repos = requests.get(
47 "{}/projects?search={}".format(source_api, source_project),
48 headers={"PRIVATE-TOKEN": source_pat},
49)
50source_project_number = search_repos.json()[0]["id"]
51source_branch = source_branch.split(",")
52
53
54def check_status(source_api, source_project_number, source_pat, source_branch, sha):
55 """ Get the workflow status of a CI job. """
56
57 ci_status = None
58
59 # Loop over each CI workflow at target repository and find the right commit.
60 pipelines_all = requests.get(
61 "{}/projects/{}/pipelines".format(source_api, source_project_number),
62 headers={"PRIVATE-TOKEN": source_pat},
63 ).json()
64
65 for pipeline in pipelines_all:
66
67 if pipeline["sha"] == sha:
68 assert (
69 pipeline["ref"] in source_branch
70 ), f"Bad branch {pipeline['ref']} != {source_branch}"
71 ci_status = pipeline["status"]
72 ci_web_url = pipeline["web_url"]
73 print(f"Found {sha} on branch {pipeline['ref']} with status '{ci_status}'.")
74 break
75
76 assert ci_status is not None, "Could not find workflow for this commit"
77
78 return ci_status, ci_web_url
79
80
81# Get the status of the work flow at the target repo for this commit.
82ci_status, ci_web_url = check_status(
83 source_api, source_project_number, source_pat, source_branch, sha
84)
85
86# Sometimes it takes a bit of time to update the status, this does a few checks.
87count = 0
88while ci_status == "running":
89 if count >= 10:
90 break
91
92 time.sleep(10)
93 ci_status, ci_web_url = check_status(
94 source_api, source_project_number, source_pat, source_branch, sha
95 )
96 count += 1
97
98
99# Convert to allowed GitHub Actions status tag.
100if ci_status == "running":
101 ci_status = "pending"
102elif ci_status == "success":
103 ci_status = "success"
104else:
105 ci_status = "error"
106
107# Append workflow to GitHub.
108status = {"state": ci_status, "target_url": ci_web_url, "context": target_repo_context}
109post_status = requests.post(
110 "{}/repos/{}/statuses/{}".format(target_api, target_project, sha),
111 headers={"Authorization": "token " + target_pat},
112 data=json.dumps(status),
113)
114
115if post_status.status_code != 201:
116 print(post_status.text)
117 exit()
The second file is the GitLab CI workflow.
1variables:
2 # Cori queue submission options.
3 SCHEDULER_PARAMETERS: "-C haswell -q debug -N1 -t 00:05:00"
4
5# How is the workflow triggered?
6workflow:
7 rules:
8 # Can only be initiated via a trigger token.
9 - if: $CI_PIPELINE_SOURCE == "trigger"
10 when: always
11
12# The CI job.
13status-repo:
14
15 # Running on Cori.
16 tags: [cori]
17
18 script:
19 - python3 status-github.py
This is very similar to the mirror
repository workflow, see above for details.