***** WARNING: THIS PAGE IS UNDER CONSTRUCTION *****

DISREGARD ITS CONTENTS UNTIL THIS WARNING IS REMOVED

This page is a simple guide to the creation of Singularity-based containers.

Containers are essential for Mufasa users, since user processes must always be run within containers. This is not just a rule imposed by Mufasa's admins, but a good practice, worth learning for HPC applications and adopted both in academia and in the enterprise.

In a nutshell, containers are complete, self-contained development environments that allow users to reproduce the exact same feature set they need on any machine, including a specific operating system, libraries, applications, and packages. They are, essentially, lightweight virtual machines you can deploy on any system featuring a container runtime (e.g., Singularity, Docker, or Podman) by simply bringing a definition file with you. A definition file is an intuitive script containing the instructions required to create (build) your container image, making it particularly convenient to migrate containers across machines and share them with your coworkers.

Though the syntax of the definition files can change depending on the specific runtime (e.g., Docker and Podman share the same syntax for their Dockerfile, while Singularity's def file varies a little), the principles and main sections are generally the same, meaning that learning how to create good quality and clear Singularity def files is a valuable skill you can reuse with any other container runtime. Likewise, the time spent learning the command syntax of Singularity to build, run, shell into, or exec commands on an interactive container or running instance (or service) is definitely worth the price.

On Mufasa, containers must be built and run via the SLURM scheduling system, either interactively or non-interactively, using Singularity as the only available container runtime.

Basic concepts

Images

In Singularity, an image is a single file representing the full container filesystem. Images can be:

a compressed file (usually, .sif files, the standard Singularity format);
or an unpacked directory for development (a sandbox).

Often, an image is based on another image, with some additional customization. For example, you can instruct Singularity to build an image starting from the Ubuntu base image, adding any other applications and configurations you need to have the resulting container provide a specific service.

You might create your own images or use those created by others and published in a registry. To build your own image, you first create a definition file, using a simple syntax that defines the steps needed to create the image; then you build the image from this file and run it.

In general, it is possible to build an image on any machine (e.g., your laptop) and then move it to another machine featuring Singularity as a container runtime, including Mufasa. However, since containers you wish to complement with GPU support require access to the Nvidia libraries installed on the system where you want to run them, it is recommended to build images needing GPU resources on Mufasa itself.

Containers

A Singularity container is simply a filesystem image being executed. Unlike Docker, Singularity containers:

are unprivileged by default and run as the calling user (not as the host's root), which makes them safe and practical in multi-user environments. More about this later;
if run from SIF files, they cannot modify the filesystem image during execution. Any change is applied only to the container instance being executed (i.e., a temporary sandbox);
the filesystem image can be modified, instead, if they are created starting from images built as sandbox (--sandbox), made writable (--writable), and run as container-exclusive root (--fakeroot).

As a consequence, installation of software libraries in a Singularity image is possible either at building time, by including all those you need in the def file specifications, or at run time if you built the image as a sandbox and applied the above-mentioned flags when running it. The main advantage of the def file-based approach is that you can easily recreate the same image whenever you need by running a single command, while with the second approach, you can simply run commands within the container itself when it gets executed and modify it (e.g., update/install libraries and applications) interactively from the command line.

Creation of a custom Singularity image

Custom Singularity images are built using a definition file, usually stored with a .def extension (and no spaces inside the file name). This file describes how your image will be built and should be placed in an empty folder containing only the files needed to perform the building process.

For all the available specifications and options that can be provided in a def file, see [official documentation]. A minimal structure is:

Bootstrap: Specifies the base image source, which is the registry from which the base image will be pulled (e.g., docker for the Docker Hub or library for the standard repository of Singularity-native images).

From: The path to the base image in the specified repository (e.g., ubuntu:22.04 for the base image of Ubuntu, available from both docker and library repositories). Usually, you can start from a base image already including most of the libraries you need (e.g., PyTorch or TensorFlow). The last part of this path (i.e., the image name) usually takes the form name:version, as in ubuntu:22.04. If the version tag is omitted in the image name, that is assumed to be latest. As a rule of thumb, to make your def file and builds more explicit and resilient, do not omit the version tag ever.

%environment: Defines environment variables set at runtime (not at build time).

%files: Copies files from the host into the container.

%post: Commands executed inside the container during build (executed as root). E.g., in this section, you should include all commands to install the Linux and Python packages you need in the container. Also, in this section, you can also define variables needed at build time. Note that all the commands issued at build time should not involve interaction with the user, since such interaction is not possible (e.g., apt install <package names> should be executed with the -y option.

%runscript: Commands executed when the container is run (i.e., when using the singularity run or singularity instance run commands).

Example of a definition file to create a TensorFlow-ready image:

# Base image of the container from which we start to create our custom one
Bootstrap: docker
From: tensorflow/tensorflow:2.16.1-gpu
#From: tensorflow/tensorflow:2.16.1 #same version but without GPU support

%files
        # We copy the 'requirements.txt' file contained in the host build directory to the 
        # container.
        # This file contains all the Python libraries we wish to include in our container.
        requirements.txt

%post
        # Set the desired Python version (environment variable)
        # NB: It is suggested to set the same version of Python already installed in the container 
        #     pulled at the beginning.
        #     For example, the container "tensorflow/tensorflow:2.16.1-gpu" runs Python 3.11
        python_version=3.11

        # Install the desired Python version and the other applications you need (the current TF 
        # image is based on Ubuntu, that's why we use <code>apt</code> as the package manager)
        apt-get update
        apt-get install -y python${python_version}-full graphviz libgl1-mesa-glx

        # Set the default Python executable on the container, so you will not need to call it in its 
        # extended form (e.g., "python3.11")
        # when executing scripts in the container.
        # Set default version for root user - modified version of this solution: 
        #     https://jcutrer.com/linux/upgrade-python37-ubuntu1810
        update-alternatives --install /usr/local/bin/python python /usr/bin/python${python_version} 1

        # Clean pip cache before updating pip
        python -m pip cache purge

        # Update pip, setuptools, and wheel
        python -m pip install --upgrade pip setuptools wheel

        # Install the Python libraries we need
        python -m pip install -r requirements.txt

        # Clean pip cache again to free up space
        python -m pip cache purge

Examples of requirements file can be found in this [| GitHub repo].

Creation of the Docker image

Once the Dockerfile is completed and all the material required by the image is in place in the work directory, it is time to create the image. The container image describes the container and can be subsequently used to create a container file: for instance, one formatted using the .sqsh format accepted by SLURM.

To create the container image, use command

docker build -t <name_of_image> .

where <name_of_image> is the name to be assigned to the image. This name is arbitrary, but usually is structured like

<name>:vXX

where <name> is any name and XX is a version number. The “.” in the docker command tells docker that the components of the container are in the current directory.

An example of command for the creation of an image is the following:

docker build -t docker_test:v1 .

During image creation, all the commands specified in the Dockerfile are executed (e.g., for the installation of libraries).

Image library

Docker maintains a local repository of (compressed) images that are available on the local computer (i.e. the one used for image creation). Every time a new image is created on the machine, it gets automatically added to the local repository.

To get a list of available images in the local repository, use command

docker image list

The local repository is in a system directory managed by Docker.

In addition to local repositories, Docker allows access to remote repositories; the main one among these is Docker Hub.

Creation of a Docker container from an image

In order to be run on Mufasa, Docker containers must take the form of a single .sqsh compressed file (it's pronounced "squash").

Creation of a .sqsh container file can be done from a local or remote image. Command

enroot import docker://<remote_container_image>

creates a container file called

<remote_container_image>.sqsh

from a remote Docker image called <remote_container_image> downloaded from the Dockerhub remote image repository.

Example:

enroot import docker://python:3.6

To create a Docker container from a local Docker image (i.e., one stored in the local image library on your own computer) the command to use is instead

enroot import dockerd://<local_container_image>

(note that we are now using import dockerd instead of import docker).

Running the command above creates a container file called

<local_container_image>.sqsh

from a local image called <local_container_image>.

Example:

enroot import dockerd://docker_test:v1

Singularity

Contents