Singularity

***** WARNING: THIS PAGE IS UNDER CONSTRUCTION *****

DISREGARD ITS CONTENTS UNTIL THIS WARNING IS REMOVED

This page is a simple guide to the creation of Singularity-based containers.

Containers are essential for Mufasa users, since user processes must always be run within containers. This is not only due to the way Mufasa is set up, but also a good practice worth learning for HPC applications and adopted both in academia and in the enterprise.

In a nutshell, containers are complete, self-contained development environments that allow users to reproduce the exact same feature set they need on any machine, including a specific operating system, libraries, applications, and packages. They are, essentially, lightweight virtual machines you can deploy on any system featuring a container runtime (e.g., Singularity, Docker, or Podman) by simply bringing a definition file with you. A definition file is an intuitive script containing the instructions required to create (build) your container image, making it particularly convenient to migrate containers across machines and share them with your coworkers.

Though the syntax of the definition files can change depending on the specific runtime (e.g., Docker and Podman share the same syntax for their Dockerfile, while Singularity's def file varies a little), the principles and main sections are generally the same, meaning that learning how to create good quality and clear Singularity def files is a valuable skill you can reuse with any other container runtime. Likewise, the time spent learning the command syntax of Singularity to build, run, shell into, or exec commands on an interactive container or running instance (or service) is definitely worth the price.

On Mufasa, containers must be built and run via the SLURM scheduling system, either interactively or non-interactively, using Singularity as the only available container runtime.

Basic concepts

Images

In Singularity, an image is a single file representing the full container filesystem. Images can be:

• a compressed file (usually, .sif files, the standard Singularity format);
• or an unpacked directory for development (a sandbox).

Often, an image is based on another image, with some additional customization. For example, you can instruct Singularity to build an image starting from the Ubuntu base image, adding any other applications and configurations you need to have the resulting container provide a specific service.

You might create your own images or use those created by others and published in a registry. To build your own image, you first create a definition file, using a simple syntax that defines the steps needed to create the image; then you build the image from this file and run it.

In general, it is possible to build an image on any machine (e.g., your laptop) and then move it to another machine featuring Singularity as a container runtime, including Mufasa. However, since containers you wish to complement with GPU support require access to the Nvidia libraries installed on the system where you want to run them, it is recommended to build images needing GPU resources on Mufasa itself.

Mufasa provides a specific QOS for users that need to build an image.

Containers

A Singularity container is simply a filesystem image being executed. Singularity containers:

• are unprivileged by default, i.e. they run with the same privileges possessed by the calling user (instead of those of the host's root), which makes them safe and practical in multi-user environments; (more about this later)
• if run from SIF files, they cannot modify the filesystem image during execution. Any change is applied only to the container instance being executed (i.e., a temporary sandbox);
• the filesystem image can be modified, instead, if they are created starting from images built as sandbox (--sandbox), made writable (--writable), and run as container-exclusive root (--fakeroot).

These features make Singularity much more suitable for an HPC environment such as Mufasa with respect to many alternatives (such as Docker).

Mufasa does not include software libraries: to use them, users need to install the libraries in the same Singularity image where the user software runs.

Installation of software libraries in a Singularity image is possible either at building time, by including all those you need in the def file specifications, or at run time if you built the image as a sandbox and applied the above-mentioned flags when running it. The main advantage of the def file-based approach is that you can easily recreate the same image whenever you need by running a single command, while with the second approach, you can simply run commands within the container itself when it gets executed and modify it (e.g., update/install libraries and applications) interactively from the command line.

Creation of a custom Singularity image

Custom Singularity images are built using a definition file, usually stored with a .def extension (and no spaces inside the file name). This file describes how your image will be built and should be placed in an empty folder containing only the files needed to perform the building process.

For all the available specifications and options that can be provided in a def file, see the official documentation. A minimal structure is:

Bootstrap

Specifies the base image source, which is the registry from which the base image will be pulled (e.g., docker for the Docker Hub or library for the standard repository of Singularity-native images).

From

The path to the base image in the specified repository (e.g., ubuntu:22.04 for the base image of Ubuntu, available from both docker and library repositories). Usually, you can start from a base image already including most of the libraries you need (e.g., PyTorch or TensorFlow). The last part of this path (i.e., the image name) usually takes the form name:version, as in ubuntu:22.04. If the version tag is omitted in the image name, that is assumed to be latest.

Suggestion: to make your def file and builds more explicit and resilient, never omit the version tag.

%environment

Defines environment variables set at runtime (not at build time).

%files

Copies files from the host into the container.

%post

Commands executed inside the container during build (executed as root). E.g., in this section, you should include all commands to install the Linux and Python packages you need in the container. Also, in this section, you can also define variables needed at build time.

Note: commands issued at build time should not involve interaction with the user, since such interaction is not possible (e.g., apt install <package names> should be executed with the -y option.

%runscript

Commands executed when the container is run (i.e., when using the singularity run or singularity instance run commands).

In a definition file, lines beginning with # are comments.

Example of definition file

This is an example definition file to create a TensorFlow-ready image. Lines starting with # are comments.

# Base image of the container from which we start to create our custom one

Bootstrap: docker

From: tensorflow/tensorflow:2.16.1-gpu

# Note: if you do not need GPU support, you can use this alternative path instead: tensorflow/tensorflow:2.16.1

%files

# We copy the 'requirements.txt' file contained in the host build directory to the container.

# This file contains all the Python libraries we wish to include in our container.

requirements.txt

%post

# Set the desired Python version (environment variable)

# Note: it is suggested to set the same version of Python already installed in the container pulled at the beginning. For example, the container "tensorflow/tensorflow:2.16.1-gpu" runs Python 3.11

python_version=3.11

# Install the desired Python version and the other applications you need (the current TF image is based on Ubuntu, that's why we use apt as the package manager)

apt-get update

apt-get install -y python${python_version}-full graphviz libgl1-mesa-glx

# Set the default Python executable on the container, so you will not need to call it in its extended form (e.g., "python3.11") when executing scripts in the container.

# Set default version for root user - modified version of this solution: https://jcutrer.com/linux/upgrade-python37-ubuntu1810

update-alternatives --install /usr/local/bin/python python /usr/bin/python${python_version} 1

# Clean pip cache before updating pip

python -m pip cache purge

# Update pip, setuptools, and wheel

python -m pip install --upgrade pip setuptools wheel

# Install the Python libraries we need

python -m pip install -r requirements.txt

# Clean pip cache again to free up space

python -m pip cache purge

Examples of requirements files can be found in this GitHub.

Creation of the Docker image

Once the Dockerfile is completed and all the material required by the image is in place in the work directory, it is time to create the image. The container image describes the container and can be subsequently used to create a container file: for instance, one formatted using the .sqsh format accepted by SLURM.

To create the container image, use command

docker build -t <name_of_image> .

where <name_of_image> is the name to be assigned to the image. This name is arbitrary, but usually is structured like

<name>:vXX

where <name> is any name and XX is a version number. The “.” in the docker command tells docker that the components of the container are in the current directory.

An example of command for the creation of an image is the following:

docker build -t docker_test:v1 .

During image creation, all the commands specified in the Dockerfile are executed (e.g., for the installation of libraries).

Image library

Docker maintains a local repository of (compressed) images that are available on the local computer (i.e. the one used for image creation). Every time a new image is created on the machine, it gets automatically added to the local repository.

To get a list of available images in the local repository, use command

docker image list

The local repository is in a system directory managed by Docker.

In addition to local repositories, Docker allows access to remote repositories; the main one among these is Docker Hub.

Creation of a Docker container from an image

In order to be run on Mufasa, Docker containers must take the form of a single .sqsh compressed file (it's pronounced "squash").

Creation of a .sqsh container file can be done from a local or remote image. Command

enroot import docker://<remote_container_image>

creates a container file called

<remote_container_image>.sqsh

from a remote Docker image called <remote_container_image> downloaded from the Dockerhub remote image repository.

Example:

enroot import docker://python:3.6

To create a Docker container from a local Docker image (i.e., one stored in the local image library on your own computer) the command to use is instead

enroot import dockerd://<local_container_image>

(note that we are now using import dockerd instead of import docker).

Running the command above creates a container file called

<local_container_image>.sqsh

from a local image called <local_container_image>.

Example:

enroot import dockerd://docker_test:v1