Containerized development environment#

The cotainr project provides a containerized development environment that can be used to develop and test cotainr itself. The containerized development environment is defined in the Dockerfile included in the DeiC-HPC/cotainr repository.

Developers that are part of the DeiC-HPC organization may pull the development containers from the orgs/DeiC-HPC GitHub Container Registry (GHCR). This requires logging into GHCR using the docker login / podman login CLI tool with a personal access token (classic) that has at least the read:packages scope. Using the GitHub gh CLI tool this may be achieved by running:

$ gh auth login --scopes read:packages
$ gh auth token | podman login ghcr.io --username $(gh api user --jq '.login') --password-stdin

Replace podman with docker in the above command if you use docker instead of podman.

Automated building of the containers#

The containerized development environment containers are automatically built and pushed to GHCR using the CI_build_docker_images.yml GitHub Action workflow. This workflow uses the official Docker GitHub Actions to build the containers for all supported architectures and dependencies as defined in the single sourced dependency matrix.

The workflow is triggered on pushes to the main branch, as well as development branches starting with docker_dev_env, if the files “defining” the development environment have changed, i.e. if changes are made to the following files:

A SHA256 checksum of these files, as calculated by the hashFiles function, is used to uniquely identify the “version” of the development environment as defined by these files. The built containers are tagged with this checksum, which allows for identification of the containers that should be used in the CI/CD pipelines, i.e. the checksum of the three files on the main branch identify the containers that must be used in the CI/CD pipelines for the main branch - likewise for other branches. Additionally, the containers are tagged with the branch name (main or docker_dev_env_*) to make it easier to pull the containers for local development.

Manually building the containers#

If you do not have access to the cotainr GHCR or want to build the containers locally, you can do so using the Dockerfile. When building the containers locally, you need to provide the SINGULARITY_PROVIDER (apptainer or singularity-ce`) and SINGULARITY_VERSION build arguments to the docker build / podman build command, e.g. from the cotainr repository root directory, when using docker:

$ docker build --build-arg SINGULARITY_PROVIDER=apptainer --build-arg SINGULARITY_VERSION=1.3.6 -t cotainr-dev-env:local -f .github/workflows/dockerfiles/Dockerfile .

Or when using podman:

$ podman build --format=docker --build-arg SINGULARITY_PROVIDER=apptainer --build-arg SINGULARITY_VERSION=1.3.6 -t cotainr-dev-env:local -f .github/workflows/dockerfiles/Dockerfile .

The --format=docker is needed as the Dockerfile contains instructions and bash commands that are not supported by the OCI image format.

Running in the containerized development environment#

The containerized development environment includes a singularity container runtime (apptainer or singularity-ce) and the uv Python package manager. At runtime, you need to run uv sync to install/update the Python environment to include the dependencies specified in the pyproject.toml file to get a fully working development environment.

Synchronizing the uv managed Python virtual environment is so fast and cheap that we have opted for doing it every time the container is started instead of building individual container images for all the different Python versions and dependencies specified in the single sourced dependency matrix.

We recommend running the containerized development environment rootless as a non-root user using either docker or podman. Since we are running one container runtime (apptainer / singularity-ce) inside another container runtime (docker / podman), for this to work, it is in general necessary to:

  1. Have unprivileged user namespaces enabled. Generally this means that the user.max_user_namespaces kernel parameter must be 1 (or larger) and the kernel.unprivileged_userns_clone kernel parameter must be set to 1. This is the default on most modern Linux distributions. Additionally, you must make sure that the other prerequisites for running rootless are met.

  2. Run with at least some of the docker / podman security options disabled. While it is possible to run with the --privileged flag, this disables all security features and is not recommended. Instead, we recommend looking at the suggested security options in the reference makefile and development container configurations and then try to run with as few of these options as possible.

We provide two reference methods to run the containerized development environment, one for running it in a terminal using a makefile and one for running the container as a development container integrated with an IDE.

A note on the ID of the user running inside the container#

The containerized development environment is designed to be run as a non-root user. On most Linux distributions, the default non-root user has user ID 1000. So in the reference makefile and development container configurations, we assume that you are running as user 1000 and map this user to the user inside the container to avoid permission issues when accessing files on the host system from inside the container.

On the GitHub action runners, used in the CI/CD pipelines, the default user has user ID 1001. Consequently, we specify --user 1001 when running the containerized development environment in the GitHub Actions workflows to avoid subtle permission errors when accessing files on the host system from inside the container, e.g. accessing the repository clone done by the checkout action. While this rootless setting should theoretically also work for running apptainer / singularity-ce inside the container on the GitHub action runners, when we need to run apptainer / singularity-ce in the container on GitHub action runners, we run as the root user with the --privileged flag. This is because it is currently too tedious to disable the apparmor restriction on kernel.unprivileged_userns_clone on GitHub action runners which seems to be necessary for running as a non-root user.

Using the reference makefile#

We provide a reference Makefile that includes targets for using the containerized development environment to run the cotainr test suite as well as build the reference documentation. It should generally work with both docker and podman, on most Linux distributions, though you may have to adjust it to your specific environment if your local user does not have user ID 1000 or if you want to limit the docker / podman security options that are disabled. Run make help for more details on the available targets.

Using the IDE development container#

We provide a reference devcontainer.json file that includes the necessary configuration to run the containerized development environment as a development container integrated with an IDE. The devcontainer.json file is mainly designed for use with Visual Studio Code with rootless podman as the container runtime. For this to work, you need to install the Dev Containers extension and set the dev.containers.dockerPath setting to podman in your Visual Studio Code settings.

The development container setup is work-in-progress

The IDE development container setup is still work-in-progress. It may not work as expected for all combinations of OS’es, IDEs, and container runtimes. It may still need further configuration to fully integrate with Visual Studio Code or other IDEs.