Quick-example pyproject.toml

An example pyprojet.toml configuration file, used for the demo code in the desc-continuous-integraion repository.

 1[build-system]
 2requires = ["setuptools >= 61.0"] # PEP 621 compliant
 3build-backend = "setuptools.build_meta"
 4
 5[project]
 6name = "mydescpackage"
 7description = "Example DESC Python package, some simple mathmatical functions."
 8readme = "README.md"
 9authors = [{ name = "Stuart McAlpine", email = "stuart.mcalpine@fysik.su.se" }]
10license = { file = "LICENCE" }
11classifiers = [
12    "Programming Language :: Python :: 3",
13]
14keywords = ["desc", "python"]
15dependencies = [
16    'numpy',
17]
18requires-python = ">=3.8"
19dynamic = ["version"] # Scrape the version dynamically from the package
20
21[tool.setuptools.dynamic]
22version = {attr = "mydescpackage._version.__version__"}
23
24[tool.setuptools.packages.find]
25where = ["src"]
26
27[project.optional-dependencies]
28ci = ["pytest", "pytest-cov", "flake8"]
29
30[project.scripts]
31display-pi = "mydescpackage.pi:display_pi"

Quick-example ci_workflow.yml

An example Continuous Integration YAML workflow (.github/workflows/ci_example_2.yml in the desc-continuous-integraion repository). More details in GitHub Actions.

 1# Author - Stuart McAlpine - stuart.mcalpine@fysik.su.se - Jan 2023
 2
 3name: DESC Example 2
 4
 5# How does the workflow get triggered?
 6on:
 7  # Triggers when push/pull-request made to the main branch.
 8  pull_request:
 9    branches:
10      - main
11  push:
12    branches:
13      - main
14
15# List of jobs for this workflow.
16jobs:
17
18  # Our pytest job.
19  ci-with-pytest:
20
21    # Our strategy lists the OS and Python versions we want to test on.
22    strategy:
23
24      # Don't quit all jobs if only one job fails.
25      fail-fast: false
26      
27      matrix:
28        python-version: ["3.8","3.9","3.10","3.11","3.12"]
29        os: [ubuntu-20.04, ubuntu-latest, macos-latest]
30        experimental: [false]
31
32        # Test on this, but don't mind if it fails.
33        include:
34          - os: ubuntu-latest
35            python-version: "3.11"
36            experimental: true
37
38    # If True, do not fail the job, just warn me.
39    continue-on-error: ${{ matrix.experimental }}
40
41    # What operating system is this job running on?
42    runs-on: ${{ matrix.os }}
43
44    # Our CI steps for this job.
45    steps:
46      # Check out this repository code.
47      - name: Check out repository code
48        uses: actions/checkout@v3
49
50      # Install Python.
51      - name: Set up Python ${{ matrix.python-version }}
52        uses: actions/setup-python@v4
53        with:
54          python-version: ${{ matrix.python-version }}
55
56      # Install my package.
57      - name: Install mydescpackage.
58        run: |
59          python -m pip install --upgrade pip
60          python -m pip install .[ci]
61          
62      # Do some basic code linting using flake8. Check for syntax and indentation errors.
63      - name: flake8 linting.
64        run: flake8 --count --select=E1,E9 --show-source --statistics ./src/mydescpackage/*.py
65
66      # Perform the unit tests and output a coverage report.
67      - name: Test with pytest
68        run: pytest --cov=mydescpackage --cov-report xml ./tests
69          
70      # Upload the code coverage results to codecov.io.
71      - name: Upload coverage to Codecov
72        uses: codecov/codecov-action@v2

NERSC CI files

Below are some details of the workings of the mirror and target repositories’ CI workflows.

Mirror repository files

The purpose of the mirror repository is to clone the contents of the source repository at GitHub to the target repository at the NERSC GitLab instance. This is done through a simple CI workflow (.gitlab-ci.yml) which runs a simple Bash script (mirror.bash), and is automatically triggered from the CI workflow at the GitHub source repository.

The two files that make up the CI workflow for the mirror repository are detailed below.

./gitlab/mirror_repo_files/mirror.bash
 1# ----------------------------------------------------------------------------#
 2# Script to clone a DESC repository from GitHub to the NERSC GitLab instance. #
 3#                                                                             #
 4# This script goes in your "mirror" repository on GitLab.                     #
 5#                                                                             #
 6# Script adapted from the CI Resources "mirroring" example                    #
 7# (https://software.nersc.gov/ci-resources/mirroring)                         #
 8#                                                                             #
 9# You need to set these in Settings->CI/CD->Variables:                        #
10#   - MIRROR_SOURCE_PAT : PAT of the source repository from GitHub.           #
11#   - MIRROR_TARGET_PAT : PAT of the target repository on GitLab.             #
12#                                                                             #
13# Other variables are passed along with the trigger token from GitHub.        #
14#                                                                             #
15# Authors:                                                                    #
16#   Stuart McAlpine (@stuartmcalpine)                                         #
17#   Heather Kelly (@heather999)                                               #
18# ----------------------------------------------------------------------------#
19
20#!/bin/bash
21
22set -e
23
24# Working directory.
25wd=$(pwd)
26
27# Always want to remove authentication file on exit (file even if workflow fails).
28function cleanup()
29{
30    echo "Cleanup: Removing ${wd}/askpass"
31    rm -rf ${wd}/askpass
32}
33trap cleanup EXIT
34
35# Set GIT_ASKPASS for git authentication.
36function set_askpass()
37{
38    echo "#!/bin/bash" > ${wd}/askpass
39    echo "echo $1" >> ${wd}/askpass
40
41    chmod +x ${wd}/askpass
42    export GIT_ASKPASS=${wd}/askpass
43}
44
45# Step 1) Fetch information from source repository.
46set_askpass ${MIRROR_SOURCE_PAT}
47echo "Cloning from ${GIBHUB_SOURCE_REPO}"
48
49rm -rf source
50git init --bare source
51
52cd source
53git remote add origin ${GITHUB_SOURCE_REPO}
54git config remote.origin.mirror true
55
56git fetch origin --prune
57
58# Step 2) Clone source repository contents to target repository.
59set_askpass ${MIRROR_TARGET_PAT}
60git remote add target ${GITLAB_TARGET_REPO}
61
62# Only pushes the MIRROR_SOURCE_BRANCH repository (and its tags).
63# Protected branches do not translate across gitlab, github, bitbucket etc.
64git push target --prune +refs/remotes/origin/$GITHUB_SOURCE_BRANCH:refs/heads/$GITHUB_SOURCE_BRANCH +refs/tags/*:refs/tags/*

The git push target --prune means we are only pushing the $GITHUB_SOURCE_BRANCH branch (and tags) of the source repository (which will likely be your main or master branch). It is possible to target multiple branches through this mechanism, however it not recommended to mirror all branches by default.

Additionally, we also use the environment variable GIT_ASKPASS to provide authentication tokens to git. This ensures that the tokens are not visible in the CLI which is an important requirement in multi-tenant hosts e.g. Cori nodes.

./gitlab/mirror_repo_files/.gitlab-ci.yml
 1variables:
 2  # Cori queue submission options.
 3  SCHEDULER_PARAMETERS: "-C haswell -q debug -N1 -t 00:05:00"
 4
 5# How is the workflow triggered?
 6workflow:
 7  rules:
 8    # Can only be initiated via a trigger token.
 9    - if: $CI_PIPELINE_SOURCE == "trigger"
10      when: always
11
12# The CI Job.
13mirror-repo:
14
15  # Running on Cori.
16  tags: [cori]
17
18  script:
19    # Script to mirror the GitHub repository.
20    - bash mirror.bash
21
22    # Trigger the GitLab CI workflow in the target repository on GitLab.
23    - >
24      curl -X POST --fail
25      --form "variables[GITLAB_STATUS_REPO]=$GITLAB_STATUS_REPO"
26      --form "variables[GITLAB_TARGET_REPO]=$GITLAB_TARGET_REPO"
27      --form "variables[GITLAB_MIRROR_REPO]=$GITLAB_MIRROR_REPO"
28      --form "variables[GITHUB_SOURCE_REPO]=$GITHUB_SOURCE_REPO"
29      --form "variables[GITHUB_SOURCE_BRANCH]=$GITHUB_SOURCE_BRANCH"
30      --form "variables[GITLAB_TARGET_PROJECT_NUMBER]=$GITLAB_TARGET_PROJECT_NUMBER"
31      --form "variables[GITLAB_MIRROR_PROJECT_NUMBER]=$GITLAB_MIRROR_PROJECT_NUMBER"
32      --form "variables[GITLAB_STATUS_PROJECT_NUMBER]=$GITLAB_STATUS_PROJECT_NUMBER"
33      --form "variables[GITLAB_STATUS_CONTEXT]=$GITLAB_STATUS_CONTEXT"
34      --form "variables[GITHUB_SHA]=$GITHUB_SHA"
35      --form token=$TARGET_TRIGGER_TOKEN
36      --form ref=$GITHUB_SOURCE_BRANCH
37      https://software.nersc.gov/api/v4/projects/$GITLAB_TARGET_PROJECT_NUMBER/trigger/pipeline

Similar to the workflows for GitHub Actions, GitLab workflows are defined in a file named .gitlab-ci.yml, which must reside in the root directory of your repository.

This workflow…

  • can only be triggered via a trigger token (which comes from the GitHub CI job).

  • runs the mirror.bash script, which clones the source repository into the target repository.

  • automatically triggers the CI workflow of the target repository, which continues the pipeline.

When submitting a CI workflow to Cori you must include the queue submission parameters, passed as the environment variable SCHEDULER_PARAMETERS. The format is the same as if you were submitting a standard job at NERSC. Our job (cloning a repository) is very quick, so we are submitting to the debug queue. However if you need a longer job runtime, or want to charge the time to a particular project, you will need to modify these parameters accordingly.

Note

The repo URLs (GITLAB_TARGET_REPO etc) and project numbers (GITLAB_TARGET_PROJECT_NUMBER etc) have been passed through along with the trigger token, originally defined in the source CI workflow.

Status repository files

The status-github.py file, shown below, is a Python script that automatically appends the CI workflow result from the target repository on GitLab to the source repository on GitHub.

./gitlab/status_repo_files/status-github.py
  1# ----------------------------------------------------------------------------#
  2# Script to report the status of the CI workflow of the target repository on  #
  3# GitLab to the source repository on GitHub.                                  #
  4#                                                                             #
  5# This script goes in your "status" repository on GitLab.                     #
  6#                                                                             #
  7# Script adapted from the CI Resources "Report Status" example                #
  8# (https://software.nersc.gov/ci-resources/report-status)                     #
  9#                                                                             #
 10# You need to set these in Settings->CI/CD->Variables:                        #
 11#   - STATUS_SOURCE_PAT : PAT of the target repository from GitLab.           #
 12#   - STATUS_TARGET_PAT : PAT of the source repository on GitHub.             #
 13#                                                                             #
 14# Other variables are passed along with the trigger token from GitHub.        #
 15#                                                                             #
 16# Authors:                                                                    #
 17#   Stuart McAlpine (@stuartmcalpine)                                         #
 18#   Heather Kelly (@heather999)                                               #
 19# ----------------------------------------------------------------------------#
 20
 21#!/usr/bin/env python3
 22import os
 23import requests
 24import json
 25import time
 26
 27# Pull out environment variables.
 28source_repo = os.getenv("GITLAB_TARGET_REPO")
 29source_branch = os.getenv("GITHUB_SOURCE_BRANCH")
 30target_repo = os.getenv("GITHUB_SOURCE_REPO")
 31source_pat = os.getenv("STATUS_SOURCE_PAT")
 32target_pat = os.getenv("STATUS_TARGET_PAT")
 33target_repo_context = os.getenv("GITLAB_STATUS_CONTEXT")
 34sha = os.getenv("GITHUB_SHA")
 35
 36# Get information from repos APIs.
 37source_api = "https://software.nersc.gov/api/v4"
 38target_api = "https://api.github.com"
 39
 40target_project = target_repo.split("/")[-2:]
 41seperator = "/"
 42target_project = seperator.join(target_project)
 43target_project = target_project.split(".")[0]
 44
 45source_project = source_repo.split("/")[-1].split(".")[0]
 46search_repos = requests.get(
 47    "{}/projects?search={}".format(source_api, source_project),
 48    headers={"PRIVATE-TOKEN": source_pat},
 49)
 50source_project_number = search_repos.json()[0]["id"]
 51source_branch = source_branch.split(",")
 52
 53
 54def check_status(source_api, source_project_number, source_pat, source_branch, sha):
 55    """ Get the workflow status of a CI job. """
 56
 57    ci_status = None
 58
 59    # Loop over each CI workflow at target repository and find the right commit.
 60    pipelines_all = requests.get(
 61        "{}/projects/{}/pipelines".format(source_api, source_project_number),
 62        headers={"PRIVATE-TOKEN": source_pat},
 63    ).json()
 64
 65    for pipeline in pipelines_all:
 66
 67        if pipeline["sha"] == sha:
 68            assert (
 69                pipeline["ref"] in source_branch
 70            ), f"Bad branch {pipeline['ref']} != {source_branch}"
 71            ci_status = pipeline["status"]
 72            ci_web_url = pipeline["web_url"]
 73            print(f"Found {sha} on branch {pipeline['ref']} with status '{ci_status}'.")
 74            break
 75
 76    assert ci_status is not None, "Could not find workflow for this commit"
 77
 78    return ci_status, ci_web_url
 79
 80
 81# Get the status of the work flow at the target repo for this commit.
 82ci_status, ci_web_url = check_status(
 83    source_api, source_project_number, source_pat, source_branch, sha
 84)
 85
 86# Sometimes it takes a bit of time to update the status, this does a few checks.
 87count = 0
 88while ci_status == "running":
 89    if count >= 10:
 90        break
 91
 92    time.sleep(10)
 93    ci_status, ci_web_url = check_status(
 94        source_api, source_project_number, source_pat, source_branch, sha
 95    )
 96    count += 1
 97
 98
 99# Convert to allowed GitHub Actions status tag.
100if ci_status == "running":
101    ci_status = "pending"
102elif ci_status == "success":
103    ci_status = "success"
104else:
105    ci_status = "error"
106
107# Append workflow to GitHub.
108status = {"state": ci_status, "target_url": ci_web_url, "context": target_repo_context}
109post_status = requests.post(
110    "{}/repos/{}/statuses/{}".format(target_api, target_project, sha),
111    headers={"Authorization": "token " + target_pat},
112    data=json.dumps(status),
113)
114
115if post_status.status_code != 201:
116    print(post_status.text)
117    exit()

The second file is the GitLab CI workflow.

./gitlab/status_repo_files/.gitlab-ci.yml
 1variables:
 2  # Cori queue submission options.
 3  SCHEDULER_PARAMETERS: "-C haswell -q debug -N1 -t 00:05:00"
 4
 5# How is the workflow triggered?
 6workflow:
 7  rules:
 8    # Can only be initiated via a trigger token.
 9    - if: $CI_PIPELINE_SOURCE == "trigger"
10      when: always
11
12# The CI job.
13status-repo:
14
15  # Running on Cori.
16  tags: [cori]
17
18  script:
19    - python3 status-github.py

This is very similar to the mirror repository workflow, see above for details.