User Jobs

This page presents the features of Mufasa that are most relevant to Mufasa's Job Users. Job Users can submit jobs for execution, cancel their own jobs, and see other users' jobs (but not intervene on them).

Job Users are by necessity SLURM users (see The SLURM job scheduling system) so you may also want to read SLURM's own Quick Start User Guide.

SLURM Partitions

Several execution queues for jobs have been defined on Mufasa. Such queues are called partitions in SLURM terminology. Each partition has features (in term of resources available to the jobs on that queue) that make the partition suitable for a certain category of jobs. SLURM command

sinfo

(link to SLURM docs) provides a list of available partitions. Its output is similar to this:

PARTITION  AVAIL  TIMELIMIT  NODES  STATE NODELIST
debug         up   infinite      1    mix gn01
small*        up   12:00:00      1    mix gn01
normal        up 1-00:00:00      1    mix gn01
longnormal    up 3-00:00:00      1    mix gn01
gpu           up 1-00:00:00      1    mix gn01
gpulong       up 3-00:00:00      1    mix gn01
fat           up 3-00:00:00      1    mix gn01

In this example, available partitions are named “debug”, “small”, “normal”, “longnormal”, “gpu”, “gpulong”, “fat”. The asterisk beside "small" indicates that this is the default partition, i.e. the one that SLURM selects to run a job when no partition has been specified.

On Mufasa, partition names have been chosen to reflect the type of job that they are dedicated to. A complete list of the features of each partition can be obtained with command

sinfo --Format=all

but its output can be overwhelming.

A less comprehensive but more readable view of partition features (can be obtained via a tailored sinfo command, i.e. one that only asks for the features that are most relevant to Mufasa users. An example of such command is this:

sinfo -o "%.10P %.6a %.4c %.17B %.54G %.11l %.11L %.4r"

Such command provides an output similar to the following:

 PARTITION  AVAIL CPUS MAX_CPUS_PER_NODE                                                   GRES   TIMELIMIT DEFAULTTIME ROOT
     debug     up   62         UNLIMITED  gpu:40gb:2(S:0-1),gpu:20gb:3(S:0-1),gpu:10gb:6(S:0-1)    infinite         n/a  yes
    small*     up   62         UNLIMITED  gpu:40gb:2(S:0-1),gpu:20gb:3(S:0-1),gpu:10gb:6(S:0-1)    12:00:00       15:00   no
    normal     up   62                24  gpu:40gb:2(S:0-1),gpu:20gb:3(S:0-1),gpu:10gb:6(S:0-1)  1-00:00:00       15:00   no
longnormal     up   62                24  gpu:40gb:2(S:0-1),gpu:20gb:3(S:0-1),gpu:10gb:6(S:0-1)  3-00:00:00     1:00:00   no
       gpu     up   62         UNLIMITED  gpu:40gb:2(S:0-1),gpu:20gb:3(S:0-1),gpu:10gb:6(S:0-1)  1-00:00:00       15:00   no
   gpulong     up   62         UNLIMITED  gpu:40gb:2(S:0-1),gpu:20gb:3(S:0-1),gpu:10gb:6(S:0-1)  3-00:00:00     1:00:00   no
       fat     up   62                48  gpu:40gb:2(S:0-1),gpu:20gb:3(S:0-1),gpu:10gb:6(S:0-1)  3-00:00:00     1:00:00   no

The columns in this output correspond to the following information (from SLURM docs), where the node is Mufasa:

%P Partition name followed by "*" for the default partition

%a State/availability of a partition

%c Number of CPUs per node accessible to the partition [for Mufasa these are the 64 CPUs minus the 2 dedicated to non-SLURM jobs]

Important! This is not the actual amount of CPUs available to jobs run on the partition! See Partition features

%B The max number of CPUs per node accessible to the partition

Important! This is not the actual amount of CPUs available to jobs run on the partition! See Partition features

%G Generic resources (gres) accessible to the partition [for Mufasa these correspond to the virtual GPUs defined with MIG]

Important! This is not the actual amount of CPUs available to jobs run on the partition! See Partition features

%l Maximum time for any job in the format "days-hours:minutes:seconds"

%L Default time for any job in the format "days-hours:minutes:seconds"

%r Only user root may initiate jobs, "yes" or "no"

Note that partitions with "yes" here are not available to Job Users

In the sinfo command used, field identifiers %... have been preceded by width specifiers in the form .N, where N is a positive integer. The specifiers define how many characters to reserve to each field in the command output, and are used to increase readability.

Partition features

The output of sinfo (see above) provides an idea of what resources are available to partitions but does not provide information about how much of these resources a partition actually makes available to the user jobs which are run on it.

The resources that each partition makes available to its jobs are in fact specified by SLURM's accounting system. These are visible using command

sacctmgr show qos format=name,priority,maxtres,maxwall -p

The output of the command is similar to the following:

Name|Priority|MaxTRES|MaxWall|
normal|200|cpu=16,gres/gpu:10gb=0,gres/gpu:20gb=0,gres/gpu:40gb=0,mem=128G|1-00:00:00|
small|500|cpu=2,gres/gpu:10gb=1,gres/gpu:20gb=0,gres/gpu:40gb=0,mem=16G|12:00:00|
longnormal|100|cpu=16,gres/gpu:10gb=0,gres/gpu:20gb=0,gres/gpu:40gb=0,mem=128G|3-00:00:00|
gpu|200|cpu=8,gres/gpu:10gb=2,gres/gpu:20gb=2,mem=64G|1-00:00:00|
gpulong|100|cpu=8,gres/gpu:10gb=2,gres/gpu:20gb=2,mem=64G|3-00:00:00|
fat|50|cpu=32,gres/gpu:10gb=2,gres/gpu:20gb=2,gres/gpu:40gb=2,mem=256G|3-00:00:00|

Its elements (unfortunately, badly formatted) are the following (for more information, see SLURM's documentation):

Name

name of the partition

Priority

priority assigned to jobs run on the partition

MaxTRES

maximum amount of resources available to jobs run on the partition

cpu is the limit on processor cores, gres/gpu:Kgb is the number of GPUs with K GB of RAM available to jobs run on the partition, mem is the amount of system RAM available to jobs run on the partition

MaxWall

maximum duration of the jobs run on the partition (after which they are killed by SLURM)

System resources subjected to usage limitations

System resources that are subjected to limitation are (these are SLURM's own terms):

cpu

the number of processor cores that a job can use

mem

the amount of RAM that a job can use

gres

the amount of “other/generic resources” that a job can use: in Mufasa, the only resources belonging to this set are the GPUs

When a resource is subjected to limitation, this means that a job cannot use arbitrary quantities of it. On the contrary, the quantity needed by the job must be requested, either by running the job on a partition for which a default amount of the resource has been defined, or by explicitly including a parameter in the srun command (or an SBATCH directive in the execution script) that executes the job via SLURM.

Partition availability

An important information that sinfo provides is the availability (also called state) of partitions. Possible partition states are:

up

the partition is available to be allocated work

drain

the partition is not available to be allocated work

down

the same as drain but the partition failed: i.e., it suffered a disruption

A partition in state drain or down requires intervention by a Job Administrator to be restored to up. Jobs waiting for that partition are paused unless the partition returns available.

Choosing the partition on which to run a job

When launching a job (as explained in Executing jobs on Mufasa) a user should select the partition that is most suitable for it according to the job's features. Launching a job on a partition avoids the need for the user to specify explicitly all of the resources that the job requires, relying instead (for unspecified resources) on the default amounts defined for the partition. Partition features explains how to find out how many of Mufasa's resources are associated to each partition.

The fact that by selecting the right partition for their job a user can pre-define the requirements of the job without having to specify them makes partitions very handy, and avoids possible mistakes. However, users can -if needed- change the resource requested by their jobs wrt the default values associated to the chosen partition. Any element of the default assignment of resources provided by a specific partition can be overridden by specifying an option when launching the job, so users are not forced to accept the default value. However, it makes sense to choose the most suitable partition for a job in the first place, and then to specify the job's requirements only for those resources that have an unsuitable default value.

Resource requests by the user launching a job can be both lower and higher than the default value of the partition for that resource. However, they cannot exceed the maximum value that the partition allows for requests of such resource, if set. If a user tries to run on a partition a job that requests a higher value of a resource than the partition‑specified maximum, the run command is refused.

In general, the larger the fraction of system resources that a job asks for, the heavier the job becomes for Mufasa's limited capabilities. Since SLURM prioritises lighter jobs over heavier ones (in order to maximise the number of completed jobs) it is a very bad idea for a user to ask for their job more resources than it actually needs: this, in fact, witl have the effect of delaying (possibly for a long time) job execution.

One of the most important resources provided to jobs by partitions is time, in the sense that a job is permitted to run for no longer than a predefined tmaximum. Jobs that exceed their allotted time are killed by SLURM.

gres syntax

Whenever it is necessary to specify the quantity of gres resources, a special syntax must be used. Since in Mufasa gres resources are used for GPUs, this syntax applies to GPUs. The name of each GPU takes the form

Name:Type

where Name is gpu for all GPUs, and Type takes the following values (defined in /etc/slurm/gres.conf):

• 40gb for GPUs with 40 Gbytes of onboard RAM
• 20gb for GPUs with 20 Gbytes of onboard
• 10gb for GPUs with 10 Gbytes of onboard RAM

When asking for a gres resource (e.g., in an srun command or an SBATCH directive of an execution script), the syntax required by SLURM is

<Name>:<Type>:<quantity>

where quantity is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of the 20 GB class the syntax is

gpu:20gb:2

Executing jobs on Mufasa

The main reason for a user to interact with Mufasa is to execute jobs that require resources not available to standard desktop-class machines. Therefore, launching jobs is the most important operation for Mufasa users: what follows explains how it is done.

Considering that all computation on Mufasa must occur within Docker containers, the jobs run by Mufasa users are always containers except for menial, non-computationally intensive jobs. The process of launching a user job on Mufasa involves two steps:

---

---

Step 1

Use SLURM to run the Docker container where the job will take place

Step 2

Launch the job from within the Docker container

---

---

As an optional preparatory step, it is often useful to define an execution script to simplify the launching process and reduce the possibility of mistakes.

The commands that SLURM provides to run jobs are

srun <options> <path_of_the_program_to_be_run_via_SLURM>

and

srun <options> <path_of_the_program_to_be_run_via_SLURM>

(see SLURM documentation: srun, sbatch). The main difference between srun and sbatch is that the first locks the shell from which it has been launched, so it is only really suitable for processes that use the console for interaction with their user; sbatch, on the other side, does not lock the shell and simply adds the job to the queue, but does not allow the user to interact with the process while it is running.

Among the options available for srun and sbatch, one of the most important is --gres=gpu:<10|20|40>gb:K, where K is an integer between 1 and the maximum number of GPUs of that type available to the partition (see gres syntax). This option specifies how many of the GPUs the program requests for use. Since GPUs are the most scarce resources of Mufasa, this option must always be explicitly specified when running a job that requires GPUs.

As already explained, a quick way to define the set of resources that a program will have access to is to use option --p <partition name>. This option specifies that SLURM will run the program on a specific partition, and therefore that it will have access to all and only the resources available to that partition. As a consequence, all options that define how many resources to assign the job, such as ‑‑gres=gpu:10gb:2, will only be able to provide the job with resources that are available to the chosen partition. Jobs that require resources that are not available to the chosen partition do not get executed.

For instance, running

srun -p small ./my_program

makes SLURM run my_program on the partition named “small”. Running the program this way means that the resources associated to this partition will be available to it for use.

Using SLURM to run a Docker container

The first step to run a user job on Mufasa is to run the Docker container where the job will take place. A container is a “sandbox” containing the environment where the user's application operates. Parts of Mufasa's filesystem can be made visible (and writable, if they belong to the user's /home directory) to the environment of the container. This allows the containerized user application to read from, and write to, Mufasa's filesystem: for instance, to read data and write results.

Each user is in charge of preparing the Docker container(s) where the user's jobs will be executed. In most situations the user can simply select a suitable ready-made container from the many which are already available for use.

In order to run a Docker container via SLURM, a user must use a command similar to the following:

srun ‑p <partition_name> ‑‑container-image <container_path.sqsh> ‑‑no‑container‑entrypoint ‑‑container‑mounts=<mufasa_dir>:<docker_dir> ‑‑gres=<gpu_resources> ‑‑mem=<mem_resources> ‑‑cpus‑per‑task <cpu_amount> ‑‑pty ‑‑time=<hh:mm:ss> <command_to_run_within_container>

All parts of the command above that come after srun are options that specify what to execute and how. Below these options are explained.

‑p <partition_name>

specifies the resource partition on which the job will be run.

Important! If ‑‑p <partition_name> is used, options that specify how many resources to assign to the job (such as ‑‑mem=<mem_resources>, ‑‑cpus‑per‑task <cpu_number> or ‑‑time=<hh:mm:ss>) can be omitted, greatly simplyfying the command. If an explicit amount is not requested for a given resource, the job is assigned the default amount of the resource (as defined by the chosen partition). A notable exception to this rule concerns option ‑‑gres=<gpu_resources>: GPU resources, in fact, must always be explicitly requested with option ‑‑gres, otherwise no access to GPUs is granted to the job.

‑‑container-image=<container_path.sqsh>

specifies the container to be run

‑‑no‑container‑entrypoint

specifies that the entrypoint defined in the container image should not be executed (ENTRYPOINT in the Dockerfile that defines the container). The entrypoint is a command that gets executed as soon as the container is run: option ‑‑no‑container‑entrypoint is useful when the user is not sure of the effect of such command.

‑‑container‑mounts=<mufasa_dir>:<docker_dir>

specifies what parts of Mufasa's filesystem will be available within the container's filesystem, and where they will be mounted; for instance, if <mufasa_dir>:<docker_dir> takes the value /home/mrossi:/data this tells srun to mount Mufasa's directory /home/mrossi in position /data within the filesystem of the Docker container. When the docker container reads or writes files in directory /data of its own (internal) filesystem, what actually happens is that files in /home/mrossi get manipulated instead. /home/mrossi is the only part of the filesystem of Mufasa that is visible to, and changeable by, the Docker container.

‑‑gres=<gpu_resources>

specifies what GPUs to assign to the container. gpu_resources must take the form --gres=gpu:<10|20|40>gb:K, where K is an integer between 1 and the maximum number of GPUs of that type available to the partition (see gres syntax). For instance, <gpus> may be gpu:40gb:2, that corresponds to giving the job control to 2 entire large‑size GPUs.

Important! The ‑‑gres parameter is mandatory if the job needs to use the system's GPUs. Differently from other resources (where unspecified requests lead to the assignment of a default amount of the resource), GPUs must always be explicitly requested with ‑‑gres.

‑‑mem=<mem_resources>

specifies the amount of RAM to assign to the container; for instance, <mem_resources> may be 200G

‑‑cpus-per-task <cpu_amount>

specifies how many CPUs to assign to the container; for instance, <cpu_amount> may be 2

‑‑pty

specifies that the job will be interactive (this is necessary when <command_to_run_within_container> is /bin/bash: see Running interactive jobs via SLURM)

‑‑time=<d-hh:mm:ss>

specifies the maximum time allowed to the job to run, in the format days-hours:minutes:seconds, where days is optional; for instance, <d-hh:mm:ss> may be 72:00:00

<command_to_run_within_container>

the executable that will be run within the Docker container as soon as it is operative.

A typical value for <command_to_run_within_container> is /bin/bash. This instructs srun to open an interactive shell session (i.e. a command-line terminal interface) within the container, from which the user will then run their job. Another typical value for <command_to_run_within_container> is python, which launches an interactive Python session from which the user will then run their job. It is also possible to use <command_to_run_within_container> to launch non-interactive programs.

Nvidia Pyxis

Some of the options described below are specifically dedicated to Docker containers: these are provided by the Nvidia Pyxis package that has been installed on Mufasa as an adjunct to SLURM. Pyxis allows unprivileged users (i.e., those that are not administrators of Mufasa) to execute containers and run commands within them. More specifically, options ‑‑container-image, ‑‑no‑container‑entrypoint, ‑‑container-mounts are provided to srun by Pyxis.

Launching a user job from within a Docker container

Once the Docker container (run as explained here) is up and running, the user is dropped to the interactive environment specified by <command_to_run_within_container>. This interactive environment can be, for instance, a bash shell or the interactive Python mode. Once inside the interactive environment, the user can simply run the required program in the usual way (depending on the type of environment).

Running interactive jobs via SLURM

As explained, SLURM command srun is suitable for launching interactive user jobs, i.e. jobs that use the terminal output and the keyboard to exchange information with a human user. If a user needs this type of interaction, they must run a bash shell (i.e. a terminal session) with a command similar to

srun --pty /bin/bash

and subsequently use the bash shell to run the interactive program. To close the SLURM-spawned bash shell, run (as with any other shell)

exit

Of course, also the “base” shell (i.e. the one that opens when an SSH connection to Mufasa is established) can be used to run programs: however, programs launched this way are not being run via SLURM and are not able to access most of the resources of the machine (in particular, there is no way to make GPUs accessible to them, and they can only access 2 CPUs). On the contrary, running programs with srun or sbatch ensures that they can access all the resources managed by SLURM.

As usual, GPU resources (if needed) must always be requested explicitly with parameter --gres=gpu:<10|20|40>gb:K, where K is an integer between 1 and the maximum number of GPUs of that type available to the partition (see gres syntax). For instance, in order to run an interactive program which needs one GPU we may first run a bash shell via SLURM with command

srun --gres=gpu:10gb:1 --pty /bin/bash

an then run the interactive program from the newly opened shell.

An alternative to explicitly specifying what resources to assign to the bash shell run via SLURM is to run /bin/bash on one of the available partitions. For instance, to run the shell on partition “small” the command is

srun -p small --pty /bin/bash

Mufasa is configured to show, as part of the command prompt of a bash shell run via SLURM, a message such as (SLURM ID xx) (where xx is the ID of the /bin/bash process within SLURM). When you see this message, you know that the bash shell you are interacting with is a SLURM-run one.

Another way to know if the current shell is the “base” shell or one run via SLURM is to execute command

echo $SLURM_JOB_ID

If no number gets printed, this means that the shell is the “base” one. If a number is printed, it is the SLURM job ID of the /bin/bash process.

Detach from a running job with screen

A consequence of the way srun operates is that if you launch an interactive job but do not plan to keep the SSH connection to Mufasa open (or if you fear that the timeout on SSH connections will cut your contact with the shell) you should use command srun inside a screen session (often simply called "a screen"), then detach from the screen (here is one of many tutorials about screen available online). Now you can disconnect from Mufasa; when you need to reach your job again, you can can reopen an SSH connection to Mufasa and then reconnect to the screen.

More specifically, to create a screen session and run a job in it:

• Connect to Mufasa with SSH
• From the Mufasa shell, run

screen

• In the screen session thus created (it has the look of an empty shell), launch your job with srun
• Detach from the screen session with ctrl + A followed by D: you will come back to the original Mufasa shell, while your process will go on running in the screen session
• You can now close the SSH connection to Mufasa without damaging your process

Later, when you are ready to resume contact with your running process:

• Connect to Mufasa with SSH
• In the Mufasa shell, run

screen -r

• You are now back to the screen session where you launched your job

• When you do not need the screen containing your job anymore, destroy it by using (within the screen) ctrl + A followed by \ (i.e., backslash)

A use case for screen is writing your program in such a way that it prints progress advancement messages as it goes on with its work. Then, you can check its advancement by periodically reconnecting to the screen where the program is running and reading the messages it printed.

Using execution scripts to run jobs

Previous Sections of this page explained how to use SLURM to run user jobs directly, i.e. by specifying the value of SLURM parameters directly on the command line.

In general, though, it is preferable to wrap the commands that run jobs into execution scripts. An execution script makes specifying all required parameters easier, makes errors in configuring such parameters less likely, and -most importantly- can be reused for other jobs.

An execution script is a Linux shell script composed of two parts:

1. a preamble, where the user specifies the values to be given to parameters, each preceded by the keyword SBATCH
2. one or more srun commands that launch jobs with SLURM using the parameter values specified in the preamble

An execution script is a special type of Linux bash script. A bash script is a file that is intended to be run by the bash command interpreter. In order to be acceptable as a bash script, a text file must:

• have the “executable” flag set
• have #!/bin/bash as its very first line

Usually, a Linux bash script is given a name ending in .sh, such as my_execution_script.sh, but this is not mandatory.

To execute the script, just open a terminal (such as the one provided by an SSH connection with Mufasa), write the scripts's full path (e.g., ./my_execution_script.sh) and press the <enter> key. The script is executed in the terminal, and any output (e.g., whatever is printed by any echo commands in the script) is shown on the terminal.

Within a bash script, lines preceded by # are comments (with the notable exception of the initial #!/bin/bash line). Use of blank lines as spacers is allowed.

Below is an example of execution script (actual instructions are shown in bold; the rest are comments):

#!/bin/bash

# ----------------start of preamble----------------

# Note: these are examples. Put your own SBATCH directives below

SBATCH --job-name=myjob

# name assigned to the job

SBATCH --cpus-per-task=1

# number of threads allocated to each task

SBATCH --mem-per-cpu=500M

# amount of memory per CPU core

SBATCH --gres=gpu:10gb:1

# number of GPUs per node

SBATCH --partition=small

# the partition to run your jobs on

SBATCH --time=0-00:01:00

# time assigned to your jobs to run (format: days-hours:minutes:seconds, with days optional)

#----------------end of preamble----------------

# ----------------srun commands-----------------

# Put your own srun command(s) below

srun ...

srun ...

# ----------------end of srun commands-----------------

As the example above shows, beyond the initial directive #!/bin/bash the script includes a series of SBATCH directives used to specify parameter values, and finally one or more srun commands that run the jobs. Any parameter accepted by commands srun and sbatch can be used as an SBATCH directive in an execution script.

Automatic job caching

When a job is run via SLURM (with or without an execution script), Mufasa exploits a (fully tranparent) caching mechanism to speed up its execution. The speedup is obtained by removing the need for the running job to execute accesses to the (mechanical and therefore relatively slow) HDDs where /home partitions reside, substituting them with accesses to (solid-state and therefore much faster) SSDs.

Each time a job is run via SLURM, this is what happens automatically:

1. Mufasa temporarily copies code and associated data from the directory where the executables are located (in the user's own /home) to a cache space located on system SSDs
2. Mufasa launches the cached copy of the user executables, using the cached copies of the data as its input files
3. The executables create their output files in the cache space
4. When the user jobs end, Mufasa copies the output files from the cache space back to the user's own /home

The whole process is completely transparent to the user. The user simply prepares the executable (or the execution script) in a subdirectory of their /home directory and runs the job. When job execution is complete, the user finds their output data in the origin subdirectory of /home, exactly as if the execution actually occurred there.

Important! The caching mechanism requires that during job execution the user does not modify the contents of the /home subdirectory where executable and data were at execution time. Any such change, in fact, will be overwritten by Mufasa at the end of the execution, when files are copied back from the caching space.

Monitoring and managing jobs

SLURM provides Job Users with several tools to inspect and manage jobs. While a Job User is able to inspect all users' jobs, they are only allowed to modify the condition of their own jobs.

From SLURM's own overview:

"User tools include

srun

(link to SLURM docs) to initiate jobs,

scancel

(link to SLURM docs) to terminate queued or running jobs,

sinfo

(link to SLURM docs) to report system status,

squeue

(link to SLURM docs) to report the status of jobs [i.e. to inspect the scheduling queue], and

sacct

(link to SLURM docs) to get information about jobs and job steps that are running or have completed.”

An example of the output of squeue is the following:

JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
  520       fat     bash acasella  R 2-04:10:25      1 gn01
  523       fat     bash amarzull  R    1:30:35      1 gn01
  522       gpu     bash    clena  R   20:51:16      1 gn01

Job state

Jobs typically pass through several states in the course of their execution. Job state is shown in column "ST" of the output of squeue as an abbreviated code (e.g., "R" for RUNNING).

The most relevant codes and states are the following:

PD PENDING

Job is awaiting resource allocation.

R RUNNING

Job currently has an allocation.

S SUSPENDED

Job has an allocation, but execution has been suspended and CPUs have been released for other jobs.

CG COMPLETING

Job is in the process of completing. Some processes on some nodes may still be active.

CD COMPLETED

Job has terminated all processes on all nodes with an exit code of zero.

Beyond these, there are other (less frequent) job states. The SLURM doc page for squeue provides a complete list of them.