CI at NERSC using GitLab

For the majority of cases the self-hosted runners provided by GitHub will be more than sufficient to test and maintain code stability for DESC repositories.

However, there are some pieces of DESC software that would benefit greatly from having the ability to deploy a CI workflow directly to the Cori and Perlmutter machines at NERSC. This allows the software to be tested more intensely within an HPC environment, gives access to specific development tools at NERSC, and gives the ability to test the code against the large datasets hosted at the facility.

As there is no way to link CI workflows using GitHub Actions to the NERSC facilities directly, we require a bit of a workaround. Starting with our source repository on GitHub, we need to create three additional supporting repositories at the NERSC GitLab instance (where we have direct access to the Cori and Perlmutter machines).

  1. A mirror repository, which clones the contents of the GitHub source repository to a target repository at GitLab.

  2. A target repository, which performs the actual CI using GitLab’s builtin CI tools (similar to GitHub Actions).

  3. A status repository, which reports the results of the CI workflow performed at the target repository back to the GitHub source repository.

This means there is a bit of manual setup required in order to perform CI at NERSC when starting from a GitHub repository, however once set up, the process is fully automated.

../../_images/workflow_diagram.png

Figure 1: Schematic of the three stages of the process described above.

Here we go over the steps required to implement a CI workflow at NERSC starting from a GitHub repository. For our demo repositories’ example workflow, the goal is the same as before, to trigger the repositories’ test suite when changes to the repositories’ codebase are made. The difference now being that these tests will be performed directly on Cori, and not on a GitHub Actions-hosted runner.

Much of this tutorial/example is templated from the CI at NERSC documentation and the CI at NERSC tutorial, which we also recommend looking at.

Note

You will need a NERSC account and access to the NERSC GitLab instance before moving forward. See here for details on how DESC members get an account at NERSC.

Getting set up

To start, you’ll need to create three repositories at your NERSC GitLab instance (replace desc-continuous-integration with the name of your repository below):

  1. A blank (“New Project -> Create Blank Project”) target repository with the same name as the GitHub source repository (e.g, desc-continuous-integration). This can be created in any namespace/group. The eventual CI workflow is performed here, therefore those needing access to the CI logs should have this repository visible to them.

  2. An imported (“New Project -> Import Project -> GitLab Export”) mirror repository with the name mirror-desc-continuous-integration. This repository must be created in your private namespace. The GitLab export file to upload is in the demo repository under ./examples/nersc_gitlab/repo_templates/.

  3. An imported (“New Project -> Import Project -> GitLab Export”) status repository with the name status-desc-continuous-integration. This repository must be created in your private namespace. The GitLab export file to upload is in the demo repository under ./examples/nersc_gitlab/repo_templates/.

../../_images/create_blank_repo.png

Figure 2: Example of creating a blank repository on GitLab.

../../_images/three_repos.png

Figure 3: At the end you should have three repositories on GitLab.

For those interested in the inner workings of the mirror and status repositories, have a look at NERSC CI files in the Appendix.

Personal Access tokens

Personal access tokens (PATs) are an alternative to using passwords for authentication to GitHub or GitLab when using the API or the command line. We need to set up PATs between our repositories in order for them to communicate securely through our CI pipeline.

  1. In the GitHub source repository, in your user profile, go to Settings -> Developer settings -> Personal Access Tokens -> Generate New Token. Name the PAT “NERSC CI”, tick “workflow”, and generate the token. Copy the generated PAT and add it as a CI/CD variable in the mirror repository called MIRROR_SOURCE_PAT (Settings -> CI/CD -> Variables). Now add the same PAT as a CI/CD variable to the status repository called STATUS_TARGET_PAT (make sure to tick “masked” when adding both variables).

  2. In the GitLab target repository, create a PAT by going to Settings -> Access Tokens. Name the token “mirror-repo”, chose an expiration date, chose the role “Maintainer”, and tick all four checkboxes. Generate the token, and add it as a CI/CD variable in the mirror repository named MIRROR_TARGET_PAT. Now add the same PAT as a CI/CD variable to the status repository called STATUS_SOURCE_PAT (again, make sure to tick masked for each).

Note

The reason the mirror and status repositories should be created in your private namespace, so only you have access, is to protect the PATs stored within them. Do not share these tokens with anyone, this is the equivalent of password sharing.

Note

Always mask PATs stored as CI/CD variables. This prevents them from being displayed within the CI workflow output.

Note

PATs can have an expiration date, you may have to periodically create new PATs.

Trigger tokens

In order for our CI pipeline to work seamlessly behind the scenes, we want the various CI workflows for each intermediate repository to trigger automatically when the previous one completes. This is done through “Trigger Tokens”, which allow us to remotely trigger CI workflows within our repositories from an external source.

  1. In the mirror repository, go to Settings -> CI/CD -> Pipeline triggers and create a trigger token with the description “trigger-from-github”. Copy the created trigger token and add it as a Secret called MIRROR_TRIGGER_TOKEN in the source repository on GitHub (Settings -> Secrets -> Actions).

  2. Go to the target repository and create a trigger token called “trigger-from-mirror”. Add this one to the mirror repository as a CI/CD variable called TARGET_TRIGGER_TOKEN.

  3. Go to the status repository and create a trigger token called “trigger-from-target”. Add this one to the target repository as a CI/CD variable called STATUS_TRIGGER_TOKEN.

Note

Trigger tokens do not expire, but be sure to keep the variables masked.

Triggering the CI workflow pipeline from GitHub

The final file to create is a GitHub Actions CI workflow to initiate the pipeline in the source repository.

The ci_nersc_template.yml workflow in .github/workflows/ from our our demo repository gives a good example of how to do this. Note we need to pass all the environment variables we will work with in the subsequent CI workflows of the pipeline, which must be defined in the initial GitHub Actions workflow file.

Essentially, all we are doing is kick-starting the pipeline (in this example manually) by initiating the mirror repositories CI workflow. You can trigger the pipeline however you wish, however remember the examples in this tutorial only work for a single chosen branch of the repository, main by default, and that each trigger will run a full CI job at NERSC.

You must modify the repository URLs and GitLab project numbers (found under Settings -> General in GitLab) in the template workflow to your own. You don’t have to modify anything below jobs:.

ci_nersc_template.yml
 1name: NERSC template
 2
 3on: [workflow_dispatch]
 4
 5env:
 6  # URL to source repository on GitHub.
 7  GITHUB_SOURCE_REPO: https://github.com/LSSTDESC/<source-repo>.git
 8  # Branch to work with.
 9  GITHUB_SOURCE_BRANCH: main
10  
11  # URL to target repository on GitLab and its project number.
12  GITLAB_TARGET_REPO: https://software.nersc.gov/<namespace>/<target-repo>.git
13  GITLAB_TARGET_PROJECT_NUMBER: <target-repo-project-number>
14  
15  # URL to mirror repository on GitLab and its project number.
16  GITLAB_MIRROR_REPO: https://software.nersc.gov/<namespace>/<mirror-repo>.git
17  GITLAB_MIRROR_PROJECT_NUMBER: <mirror-repo-project-number>
18  
19  # URL to status repository on GitLab, its project number, and tag.
20  GITLAB_STATUS_REPO: https://software.nersc.gov/<namespace>/<status-repo>.git
21  GITLAB_STATUS_PROJECT_NUMBER: <status-repo-project-number>
22  GITLAB_STATUS_CONTEXT: NERSC
23  
24jobs:
25  trigger-mirror-repo-ci:
26  
27    runs-on: ubuntu-latest
28    
29    steps:
30      # Trigger mirror repository CI workflow, passing variables as we go.
31      - run: >-
32           curl -X POST --fail 
33           --form "variables[GITLAB_STATUS_REPO]=$GITLAB_STATUS_REPO"
34           --form "variables[GITLAB_TARGET_REPO]=$GITLAB_TARGET_REPO"
35           --form "variables[GITLAB_MIRROR_REPO]=$GITLAB_MIRROR_REPO"
36           --form "variables[GITHUB_SOURCE_REPO]=$GITHUB_SOURCE_REPO"
37           --form "variables[GITHUB_SOURCE_BRANCH]=$GITHUB_SOURCE_BRANCH"
38           --form "variables[GITLAB_TARGET_PROJECT_NUMBER]=$GITLAB_TARGET_PROJECT_NUMBER"
39           --form "variables[GITLAB_MIRROR_PROJECT_NUMBER]=$GITLAB_MIRROR_PROJECT_NUMBER"
40           --form "variables[GITLAB_STATUS_PROJECT_NUMBER]=$GITLAB_STATUS_PROJECT_NUMBER"
41           --form "variables[GITLAB_STATUS_CONTEXT]=$GITLAB_STATUS_CONTEXT"
42           --form "variables[GITHUB_SHA]=${{ github.sha }}"
43           --form token=${{ secrets.MIRROR_TRIGGER_TOKEN }}
44           --form ref=main
45           https://software.nersc.gov/api/v4/projects/$GITLAB_MIRROR_PROJECT_NUMBER/trigger/pipeline

Building a GitLab CI workflow for your repository

Now that we are set up, we can think about how to implement a CI workflow at GitLab. As mentioned previously, the CI workflows for GitLab are placed in a file called .gitlab-ci.yml, which is located in the root directory of your source repository. Here we will cover how to build a basic CI workflow with GitLab, relating back to the GitHub Actions syntax we learned previously. A more in-depth look at GitLab’s CI syntax can be found in The Complete GitLab CI Reference Guide.

.gitlab-ci.yml
 1variables:
 2  # Cori queue submission options.
 3  SCHEDULER_PARAMETERS: "-C haswell -M escori -q xfer -N1 -t 01:00:00"
 4
 5# How is this workflow triggered?
 6workflow:
 7  rules:
 8    # Can only be initiated via a trigger token.
 9    - if: $CI_PIPELINE_SOURCE == "trigger"
10      when: always
11
12# The CI job.
13example:
14
15  # Running on Cori.
16  tags: [cori]
17
18  script:
19    # Script loads desc-python Conda environment and runs tests.
20    - bash examples/nersc_gitlab/example-test.bash 
21
22    # Trigger the status repository CI workflow to report results.
23    - >
24      curl -X POST --fail
25      --form "variables[GITLAB_STATUS_REPO]=$GITLAB_STATUS_REPO"
26      --form "variables[GITLAB_TARGET_REPO]=$GITLAB_TARGET_REPO"
27      --form "variables[GITLAB_MIRROR_REPO]=$GITLAB_MIRROR_REPO"
28      --form "variables[GITHUB_SOURCE_REPO]=$GITHUB_SOURCE_REPO"
29      --form "variables[GITHUB_SOURCE_BRANCH]=$GITHUB_SOURCE_BRANCH"
30      --form "variables[GITLAB_TARGET_PROJECT_NUMBER]=$GITLAB_TARGET_PROJECT_NUMBER"
31      --form "variables[GITLAB_MIRROR_PROJECT_NUMBER]=$GITLAB_MIRROR_PROJECT_NUMBER"
32      --form "variables[GITLAB_STATUS_PROJECT_NUMBER]=$GITLAB_STATUS_PROJECT_NUMBER"
33      --form "variables[GITLAB_STATUS_CONTEXT]=$GITLAB_STATUS_CONTEXT"
34      --form "variables[GITHUB_SHA]=$GITHUB_SHA"
35      --form token=$STATUS_TRIGGER_TOKEN
36      --form ref=main
37      https://software.nersc.gov/api/v4/projects/$GITLAB_STATUS_PROJECT_NUMBER/trigger/pipeline

This is the GitLab CI workflow for our example repository that will eventually get run by the target repository on the NERSC GitLab instance. There are many similarities with the GitHub Actions CI workflow syntax. We must tell the workflow how to be triggered (same as on: in GitHub Actions), which is under workflow: rules:. In this case we are saying we only want the workflow to run if it was triggered via a trigger token (see the reference guide linked above for a full list of workflow options). We can set environment variables for the job, under variables:. And then we list our jobs.

As with GitHub Actions, we can have multiple jobs, each will run independently and in parallel unless you deliberately link them. This workflow only has one job, example. Within each job, tags: is the same as runs-on: from GitHub Actions. Here we want to run on Cori. And script: contains the sequence of commands to execute for the job, the same as steps: from GitHub Actions.

When running at NERSC you have to set the SCHEDULER_PARAMETERS environment variable, which defines your queue preferences for the machine (which queue to submit to, which project to charge, etc). For more details about this, and other specifics of running CI at NERSC, see here. Example repositories from NERSC for CI can also be found here.

When everything has finished successfully, you should see the NERSC CI status beside the commit on the GitHub repository main page. Clicking its “details” will take you to the GitLab CI report for the job.

../../_images/nersc_success.png

Figure 4: The NERSC tag is the reported status from the NERSC CI job.

Note

The SCHEDULER_PARAMETERS in the mirror and status repositories’ .gitlab-ci.yml files are set to SCHEDULER_PARAMETERS: "-C haswell -q debug -N1 -t 00:05:00". These jobs are extremely lightweight, which is why they goto the debug queue, however you may have to change this for your needs, for example to charge against a specific NERSC project.

Note

If your package uses submodules for some of its dependencies, you will have to add GIT_SUBMODULE_STRATEGY: recursive to the variables in the target repositories .gitlab-ci.yml.

Things to think about with CI at NERSC

Note

When deploying a CI workflow to Cori you are running code in the same environment, with the same permissions, as if you were working on a login node. Therefore things like $HOME refer to your real home directory, and you need to be careful about what scope your give to your CI workflows at NERSC. As the developer, you are responsible for the code that is run, and you need to fully understand what is happening in the workflow that you are implementing.

Note

You should not automate the mirroring of code you do not own. If you must, only clone protected branches of repositories you do not own.

Note

Any code you mirror onto the NERSC GitLab instance must adhere to the broader NERSC user policies.