User Jobs

Running jobs with SLURM

Users of Mufasa must use SLURM to run resource-heavy processes, i.e. computing jobs that require one or more of the following:

• GPUs
• multiple CPUs
• powerful CPUs
• a significant amount of RAM.

In fact, only processes run via SLURM have access to all the resources of Mufasa. Processes run outside SLURM are executed by the login server virtual machine, which has minimal resources and no GPUs. Using SLURM is therefore the only way to execute resource-heavy jobs on Mufasa. This is a key difference between Mufasa 1.0 and Mufasa 2.0.

srun and sbatch

SLURM provides two commands to run jobs, called srun and sbatch:

srun [options] <command_to_be_run_via_SLURM>

sbatch [options] <command_to_be_run_via_SLURM>

In both cases, <command_to_be_run_via_SLURM> can be any Linux program (including shell scripts). By using srun or sbatch, the command or script specified by <command_to_be_run_via_SLURM> (including any programs launched by it) are added to SLURM's execution queues.

The main difference between srun and sbatch is that the first locks the shell from which it has been launched, so it is only really suitable for interactive jobs: i.e., processes that use the console to interact with their user during job execution. sbatch, on the other side, does not lock the shell and simply adds the job to the queue, but does not allow the user to interact with the process while it is running.

sbatch provides an additional possibility: <command_to_be_run_via_SLURM> can in fact be an execution script, i.e. a special (and SLURM-specific) type of Linux shell script that includes SBATCH directives. SBATCH directives can be used to specify the values of some of the parameters that would otherwise have to be set using the [options] part of the sbatch command. This is handy because it allows to write down the parameters in an execution script instead of having to write them in the command line while launching a job, which greatly reduces the possibility of mistakes. Also, an execution script is easy to keep and reuse.

Immediately after a srun or sbatch command is launched by a user, SLURM outputs a message informing the user that the job has been queued. The output is similar to this:

srun: job 10849 queued and waiting for resources

The shell is now locked while SLURM prepares the execution of the user program (if you are using screen you can detach from that shell and come back later).

When SLURM is ready to run the program, it prints a message similar to

srun: job 10849 has been allocated resources

and then executes the program.

Options of srun and sbatch

The [options] part of srun and sbatch commands is used to tell SLURM what resources the job needs to be executed the job and how much time it will need to complete its execution.

For what concerns resources, the most important option is --qos <qos_name>, specifying which SLURM SLURM QOS the job will use. A job run with a given QOS has access to all and only the resources available to that QOS. As a consequence, all options that define how many resources to assign the job will only be able to provide the job with resources that are available to the chosen QOS. Jobs that require resources that are not available to the chosen QOS do not get executed.

If the user forgets to use option --qos <qos_name>, the job is run on the default qos (normal) which has access to zero resources. Therefore it is always necessary to specify option --qos <qos_name> when launching a SLURM job on Mufasa.

More generally, the most relevant among the [options] are:

‑-qos=<qos__name>

specifies the SLURM QOS that the job will use. It is mandatory to specify one.

Important! The chosen QOS limits the resources that can be requested, since it is not allowed to request resources (type or quantity) that exceed what is available to the chosen QOS.

Important! If ‑‑qos <qos_name> is used and options that specify how many resources to assign to the job (such as ‑‑mem=<mem_resources>, ‑‑cpus‑per‑task=<cpu_amount> or ‑‑time=<duration>) are omitted, the job is assigned the default amount of the resource (as defined by the chosen QOS. A notable exception concerns option ‑‑gres=<gpu_resources>, which is always required (see below) if the job uses a QOS with access to GPUs.

--job-name=<jobname>

Specifies a name for the job. The specified name will appear along with the JOBID number when querying running jobs on the system with squeue. The default job name (i.e., the one assigned to the job when --job-name is not used) is the executable program's name.

‑‑gres=<gpu_resources>

specifies what GPUs to assign to the job. gpu_resources is a comma-delimited list where each element has the form gpu:<Type>:<amount>, where <Type> is one of the types of GPU available on Mufasa (see gres syntax) and <amount> is an integer between 1 and the number of GPUs of such type available to the partition. For instance, <gpu_resources> may be gpu:40gb:1,gpu:3g.20gb:1, corresponding to asking for one "full" GPU and 1 "small" GPU.

Important! The ‑‑gres parameter is mandatory if the job is run with a QOS that allows access to the system's GPUs. Differently from other resources (where unspecified requests lead to the assignment of a default amount), GPUs must always be explicitly requested.

‑‑mem=<mem_resources>

specifies the amount of RAM to assign to the job; for instance, <mem_resources> may be 200G

‑‑cpus-per-task=<cpu_amount>

specifies how many CPUs to assign to the job; for instance, <cpu_amount> may be 2

‑‑time=<duration>

specifies the maximum time allowed to the job to complete, in the format days-hours:minutes:seconds, where days is optional; for instance, <d-hh:mm:ss> may be 72:00:00. When the time expires, the job (if still running) gets killed by SLURM.

‑‑pty

specifies that the job will be interactive (this is necessary when <command_to_run_within_container> is /bin/bash: see Interactive jobs)

Note that GPU resources (if needed) must always be requested explicitly. For instance, in order to execute program ./my_program which needs one GPU of type 3g.20gb with QOS gpulight we can use the SLURM command

srun --qos=gpulight --gres=gpu:3g.20gb:1 ./my_program

Interactive jobs

An interactive job is a process that use the console to interact with their user during job execution. Such a process is manually run by the user from a bash shell (i.e. a terminal session) provided by SLURM.

In order to ask SLURM to schedule the execution of a shell where the user can subsequently run the interactive job, it is necessary to use option --pty.

For instance, to ask SLURM to run a shell with QOS nogpu, the user should use command

srun --qos=nogpu --pty /bin/bash

By not specifying any other options, the user is telling SLURM that they want the shell spawned by SLURM to be provided with the default amount of resources associated to QOS nogpu. More generally, any combination of the other options of srun can be used together with --pty.

As every other job request to SLURM, the request to run a shell must be done from the login server. As soon as possible (i.e., as soon as the necessary resources are available) SLURM will open (in the same terminal that the user used to launch the srun command) a bash shell, where the user will be able to run their interactive programs.

To the user, this corresponds to the fact that the shell they were using to interact with the login server changes into a shell opened directly on Mufasa. This corresponds to the command prompt changing from

<username>@mufasa2-login:~$

to

<username>@mufasa2:~$

Another way to know if the current shell is the “base” shell or one run via SLURM is to execute command

echo $SLURM_JOB_ID

If no number gets printed, this means that the shell is the “base” one. If a number is printed, it is the SLURM job ID of the /bin/bash process.

When the user does not need the SLURM-spawned shell anymore, they should close it with command (the same used for any other Linux shell)

exit

to make the resources reserved for the interactive shell free again.

Non-interactive jobs

srun commands are very complex, and it's easy to forget some option or make mistakes while using them. For non-interactive jobs, there is a solution to this problem.

When the user job is non-interactive, in fact, the srun command can be substituted with a much simpler sbatch command. As already explained, sbatch can make use of an execution script to specify all the parts of the command to be run via SLURM. So the command becomes

sbatch <execution_script>

An execution script is a special type of Linux script that includes SBATCH directives. SBATCH directives are used to specify the values of the parameters that are otherwise set in the [options] part of an srun command.

Note on Linux shell scripts

A shell script is a text file that will be run by the bash shell. In order to be acceptable as a bash script, a text file must: have the “executable” flag set (see here for details) have #!/bin/bash as its very first line Usually, a Linux shell script is given a name ending in .sh, such as my_execution_script.sh, but this is not mandatory. Within any shell script, lines preceded by # are comments (with the notable exception of the initial #!/bin/bash line). Use of blank lines as spacers is allowed.

An execution script is a Linux shell script composed of two parts:

1. a preamble, composed of directives using which the user specifies the values to be given to parameters, each preceded by the keyword SBATCH
2. [optionally] one or more srun commands that launch jobs with SLURM using the parameter values specified in the preamble

Below is an execution script template to be copied and pasted into your own execution script text file.

The template includes all the options already described above, plus a few additional useful ones (for instance, those that enable SLURM to send email messages to the user in correspondence to events in the lifecycle of their job). Information about all the possible options can be found in [SLURM's own documentation].

In the template below, #SBATCH directives are requests made to SLURM. Notice that, though #SBATCH directives have a leading "#", that does not mean that they are comments: exactly as the #!/bin/bash at the beginning of a shell script, while starting with "#", is not a comment as well.

Other lines in the script that begin with # not followed by SBATCH are comments.

For what concerns directive that ask for a given amount of a resource (including time), if they are missing from the execution script (or commented out) the job will be assigned the default amount of the resource.

#!/bin/bash

#----------------start of preamble----------------

#SBATCH ‑-nodes=1

#SBATCH ‑‑ntasks=1

#SBATCH ‑-partition=jobs

#SBATCH ‑-qos=<qos_name>

#SBATCH ‑‑gres=<gpu_resources>

#SBATCH ‑‑mem=<mem_resources>

#SBATCH ‑‑cpus-per-task=<cpu_amount>

#SBATCH ‑‑time=<d-hh:mm:ss>

#SBATCH ‑‑output=./<filename>-%j.out

# the text file where the output of the job gets written (i.e., standard output gets redirected onto the file). "%j" is the current time.

#SBATCH ‑‑error=./<filename>-error-%j.out

# the text file where any error messages generated by the job gets written (i.e., standard error gets redirected onto the file). "%j" is the current time.

#SBATCH --job-name=<name>

# The following directives of the preamble are optional and activate SLURM's email notifications:

# the first specifies where they are sent; the following 3 set up notifications start/end/failure of job execution

#SBATCH --mail-user <email_address>

#SBATCH --mail-type BEGIN

#SBATCH --mail-type END

#SBATCH --mail-type FAIL

#----------------end of preamble----------------

<command_to_run>

<command_to_run>

...

Executing jobs on Mufasa

Key concept

The key concept about executing jobs on Mufasa is that all computation on Mufasa must occur within containers.

A container is a “sandbox” containing the environment where the user's application operates. Parts of Mufasa's filesystem can be made visible (and writable, if the user has writing permission on them: e.g., the user's /home directory) to the environment of the container. This allows the containerized user application to read from, and write to, Mufasa's filesystem: for instance, to read data and write results.

The system used by Mufasa to create and execute containers is Singularity. This wiki includes directions on preparing containers with Singularity.

The container where a user job runs must contain all the libraries needed by the job. In fact (for maintainability and safety reasons) no software and no libraries are installed on Mufasa 2.0.

Interactive and non-interactive user jobs

This section explains how to execute a user job contained in a container. It considers two types of user jobs, i.e.:

Interactive user jobs

as already explained, these are jobs that require interaction with the user while they are running, via a bash shell running within the container. The shell is used to receive commands from the user and/or print output messages. For interactive user jobs, the job is usually launched manually by the user (with a command issued via the shell) after the container is in execution.

Non-interactive user jobs

are the most common variety. The user prepares the container in such a way that, when in execution, the container autonomously puts the user's jobs into execution. The user does not have any communication with the container while it is in execution. Executing the container and running the required programs within the container's environment is done via execution scripts.

Using SLURM to run an interactive job on Mufasa

The first step to run an interactive user job on Mufasa is to run the container where the job will take place. Each user is in charge of preparing the container(s) where the user's jobs will be executed.

In order to run a container via SLURM by hand, i.e. via an interactive shell, a user must first open the shell with command

srun [general_SLURM_options] ‑‑pty /bin/bash

where [general_SLURM_options] are those already described above.

Then the user must run the container: this is done as follows.

First, it is necessary to load the Singularity software module with

module load amd/singularity

(if needed, the list of software modules available in the system can be obtained with command module av).

Then, the user must use Singularity to run the container with command (see the section about Singularity for further details)

singularity run <repository>://<name_of_container>

which pulls the container from the specified repository and executes it. Possible values for <repository> are:

docker (Docker Hub)

library (Singularityhub)

path/to/container if the container is local, i.e. located in the filesystem of Mufasa

As soon as the container is in execution, the terminal window used, so far, to interact with Mufasa becomes a shell in the container. This shell belongs to the software environment of the container, and the user can use it to interact with the container's own software environment and filesystem.

It is easy to understand if a shell is open to Mufasa or to the container because in a container shell the system prompt becomes

singularity>

Interaction between container filesystem and local filesystem

The filesystem inside the container and the local one, i.e. Mufasa's, can interact. This means that the container can access the local filesystem to read and/or write files. However, the only parts of Mufasa's filesystem that can be accessed by the container are those that the user running the container has access rights to.

As a default, the user's /home/username directory on Mufasa is automatically mapped onto /home/username into the filesystem of the container. Whatever is done to that container directory, the changes are actually applied to the local /home/username directory on Mufasa.

The mapping of the home directory does not need to be explicitly requested. However, if the user needs (in addition to the home directory) other parts of the local filesystem of Mufasa to be mapped onto the container's filesystem, this is possible by using this modified version of the singularity run command:

singularity run --bind </path/to/local/directory>:<path/to/container/directory> <repository>://<name_of_container>

If <path/to/container/directory> does not exist in the container's filesystem, it gets created by Singularity.

Using SLURM to run a non-interactive job on Mufasa

When the user job to be executed into a container is non-interactive, the mechanism based on an execution script already described in Non-interactive jobs is employed. The command to run the script which in turn will run the container where the user job takes place is therefore

sbatch <execution_script>

The general features of a SLURM execution script and the SBATCH directives used for generic jobs have already been described. Here we focus, therefore, on the SBATCH directives specifically used when SLURM is used to run a non-interactive job within a container.

Below is an execution script template to be copied and pasted into your own execution script text file.

#!/bin/bash

#----------------start of preamble----------------

#SBATCH directives already described above

#----------------end of preamble----------------

module load amd/singularity

singularity run <repository>://<name_of_container> <command_to_run>

In the last line of the script, <command_to_run> is the command (e.g., the name of an executable script), complete with path within the container's filesystem, of the program to be run into the container. Please refer to the section about Singularity for details about its commands.

The interactions between container filesystem and local filesystem in non-interactive jobs are exactly the same already described for interactive jobs. In particular, the user's home directory is mapped by default onto the filesystem of the container.

If, in addition to that, the user needs another part of the filesystem of Mufasa are to be mapped onto the container's filesystem, this is possible using this modified version of the singularity run command at the end of the script:

singularity run --bind </path/to/local/directory>:<path/to/container/directory> <repository>://<name_of_container>

If <path/to/container/directory> does not exist in the container's filesystem, it gets created by Singularity.

Job output

The whole point of running a user job is to collect its output. Usually, such output takes the form of one or more files generated within the filesystem of Mufasa by the container where the computation takes place.

As explained below, SLURM includes a mechanism to mount a part of Mufasa's own filesystem onto the container's filesystem: so when the job running within the container writes to this mounted part, it actually writes to Mufasa's filesystem. This means that when the container ends its execution, its output files persist in Mufasa's filesystem (usually in a subdirectory of the user's own /home directory) and can be retrieved by the user at a later time.

The same mechanism can be used to allow user jobs running into a container to read their input data from Mufasa's filesystem (usually a subdirectory of the user's own /home directory).

Cancelling completed jobs

When a user process run via SLURM has completed its execution and is not needed anymore, it is important to close it with scancel. Especially if much time remains to the end of the execution time requested by the job.

Cancelling a SLURM job makes the resources reserved by SLURM free again for other users, and thus speeds up the execution of the jobs still queued.

Typically, one doesn't know how long a piece of code will take to complete its work. So please make sure to check from time to time if that happened, and -if there's still time before the duration of your SLURM job ends- just scancel the job. Other users will be grateful :-)

Please note that job priority for your user depends (also) on the overall duration of the jobs that you ran on Mufasa. Therefore, cancelling jobs that are not needed anymore improves your future jobs' priority.

Detaching from a running job with screen

A consequence of the way srun operates is that if you launch an interactive user job, the shell where the command is running must remain open: if it closes, the job terminates. That shell runs in the terminal of your own PC where the SSH connection to Mufasa exists.

If you do not plan to keep the SSH connection to Mufasa open (for instance because you have to turn off or suspend your PC), there is a way to keep your interactive job alive. Namely, you should use command srun inside a screen session (often simply called "a screen"), then detach from the screen (here is one of many tutorials about screen available online).

Once you have detached from the screen session, you can close the SSH connection to Mufasa without damage. When you need to reach your (still running) job again, you can can open a new SSH connection to Mufasa and then reattach to the screen.

A use case for screen is writing your program in such a way that it prints progress advancement messages as it goes on with its work. Then, you can check its advancement by periodically reconnecting to the screen where the program is running and reading the messages it printed.

Basic usage of screen is explained below.

Creating a screen session, running a job in it, detaching from it

1. Connect to the login server with SSH
2. From the login server shell, run

   screen

3. In the screen session ("screen") thus created (it has the look of an empty shell), launch your job with srun
4. Detach from the screen by pressing ctrl + A followed by D: you will come back to the original login server shell, while your process will go on running in the screen
5. You can now close the SSH connection to the login server without damaging your running job

Reattaching to an active screen session

1. Connect to the login server with SSH
2. In the login server shell, run

   screen -r

3. You are now back to the screen where you launched your job

Closing (i.e. destroying) a screen session

When you do not need a screen session anymore:

1. reattach to the active screen session as explained above
2. destroy the screen by pressing ctrl + A followed by \ (i.e., backslash), then confirming that you really want to proceed

Of course, any program (including SLURM jobs) running within the screen gets terminated when the screen is destroyed.

Using salloc to reserve resources

What is salloc?

salloc is a SLURM command that allows a user to reserve a set of resources (e.g., a 40 GB GPU) for a given time in the future.

The typical use of salloc is to "book" an interactive session where the user enjoys complete control of a set of resources. The resources that are part of this set are chosen by the user. Within the "booked" session, any job run by the user that relies on the reserved resources is immediately put into execution by SLURM.

More precisely:

• the user, using salloc, specifies what resources they need and the time when they will need them;
• when the delivery comes, SLURM creates an interactive shell session for the user;
• within such session, the user can use srun and sbatch to run programs, enjoying full (i.e. not shared with anyone else) and instantaneous access to the resources.

Resource reservation using salloc is only possible if the request is done in advance wrt the delivery time. The more the resources that the user wants to reserve are in high demand, the more anticipated the request should be to ensure that SLURM is able to fulfill it.

When a user makes a request for resources with salloc, the request (called an allocation) gets added to the job queue of SLURM of the requisite partition as a job in pending (PD) state (job states are described here). Indeed, resource allocation is the first part of SLURM's process of executing a user job, while the second part is running the program and letting it use the allocated resources. Using salloc actually corresponds to having SLURM perform the first part of the process (resource allocation) while leaving the second part (running programs) to the user.

Until the delivery time specified by the user comes, the allocation remains in state PD, and other jobs requesting the same resources, even if submitted later, are executed. While the request waits for the delivery time, however, it accumulates a priority that increases over time. The longer the allocation stays in the PD state, the stronger this accumulation of priority: so, by requesting resources with salloc well in advance of the delivery time, users can ensure that the resources they need will be ready for them at the requested delivery time, even if these resources are highly contended.

salloc commands

salloc commands use a similar syntax to srun commands. In particular, salloc lets a user specify what resources they need and -importantly- a delivery time for the requested resources (delivery time can also be specified with srun, but in that case it is not very useful).

The typical salloc command has this form:'

salloc [general_SLURM_options] --begin=<time>

where

[general_SLURM_options]

represents the options already described in Options of srun and sbatch

--begin=<time>

specifies the delivery time of the resources reserved with salloc, according to the syntax described below. The delivery time must be a future time.

Syntax of parameter --begin

If the allocation is for the current day, you can specify <time> as hours and minutes in the form

HH:MM

If you want to specify a time of a different day, the form for is YYYY-MM-DDTHH:MM, where the uppercase 'T' separates date from time.

It is also possible to specify as relative to the current time, in one of the following forms:

now+Kminutes

now+Khours

now+Kdays

where K is a (positive) integer.

Examples:

--begin=16:00

--begin=now+1hours

--begin=now+1days

--begin=2030-01-20T12:34:00

Note that Mufasa's time zone is GMT, so <time> must be expressed in GMT as well. If you want to know Mufasa's current time, use command

date

It provides an output similar to the following:

Thu Nov 10 16:43:30 UTC 2022

How to use salloc

In the typical scenario, the user of salloc will make use of screen. Command screen creates a shell session (called "a screen") that it is possible to abandon without closing it (detaching from the screen). It is then possible to reach again the screen at a later time (reattaching to the screen). This means that a user can create a screen, run salloc within it to create an allocation for time X, detach from the screen and reattach to it just before time X to use the reserved resources from the interactive session created by salloc.

More precisely, the operations needed to do this are the following:

1. Connect to the login server.
2. From the login server shell, run

   screen

3. In the screen session ("screen") thus created run the salloc command, specifying via its options the resources you need and the time at which you want them delivered.
4. SLURM will respond with a message similar to

   salloc: Pending job allocation XXXX

5. Detach from the screen by pressing ctrl + A followed by D: you will come back to the original login server shell.
6. You can now close the SSH connection to the login server without damaging your resource allocation request.
7. At the delivery time you specified in the salloc command, connect to the login server with SSH.
8. Once you are in the login server shell, reattach to the screen with command

   screen -r

9. You are now back to the screen where you used salloc; as soon as SLURM provides to you with the resources you reserved, message "salloc: Pending job allocation XXXX" changes to the shell prompt.
10. You are now in the interactive shell session you booked with salloc. From here, you can run any programs you want, including srun and sbatch. For the whole duration of the allocation, your programs have unrestricted use of all the resources you reserved with salloc.
    Important! Any job run within the shell session is subject to the time limit (i.e., maximum duration) imposed by the partition it is running on! Therefore, if the job reaches the time limit, it gets forcibly terminated by SLURM. Termination depends exclusively from the time limit: so it occurs even if the end time for the allocation has not been reached yet. (Of course, the job also gets terminated if the allocation ends.)
11. Once the interactive shell session is not needed anymore, cancel it by exiting from the session with

    exit

    (Note that if you get to the end of the time period you specified in your request without closing the shell session, SLURM does it for you, killing any programs still running.)
12. You are now back to your screen. Destroy it by pressing ctrl + A followed by \ (i.e., backslash) to get back to the login server shell.

Cancelling a resource request made with salloc

To cancel a request for resources made as explained in How to use salloc, follow these steps:

1. Connect to the the login server with SSH.
2. Once you are in the login server shell, reattach to the screen where you used command salloc with command

   screen -r

3. You should see the message "salloc: Pending job allocation XXXX" (if the allocation is still pending) or ""salloc: job XXXX queued and waiting for resources" (if the allocation is done and waiting for its start time). Now just press Ctrl + C. This communicates to SLURM your intention to cancel your request for resources.
4. SLURM will communicate the cancellation with message

   salloc: Job allocation XXXX has been revoked.

5. Destroy the screen by pressing ctrl + A followed by \ (i.e., backslash) to get back to the login server shell.

Monitoring and managing jobs

SLURM provides Job Users with tools to inspect and manage jobs. While a Job User is able to see all users' jobs, they are only allowed to interact with their own.

The main commands used to interact with jobs are squeue to inspect the scheduling queues and scancel to terminate queued or running jobs.

Inspecting jobs with squeue

Running command

squeue

provides an output similar to the following:

JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
  520       fat     bash acasella  R 2-04:10:25      1 gn01
  523       fat     bash amarzull  R    1:30:35      1 gn01
  522       gpu     bash    clena  R   20:51:16      1 gn01

This output comprises the following information:

JOBID

Numerical identifier of the job assigned by SLURM

This identifier is used to intervene on the job, for instance with scancel

PARTITION

the partition that the job is run on

NAME

the name assigned to the job; can be personalised using the --job-name option

USER

username of the user who launched the job

ST

job state (see Job state for further information)

TIME

time that has passed since the beginning of job execution

NODES

number of nodes where the job is being executed (for Mufasa, this is always 1 as it is a single machine)

NODELIST (REASON)

name of the nodes where the job is being executed: for Mufasa it is always gn01, which is the name of the node corresponding to Mufasa.

To limit the output of squeue to the jobs owned by user <username>, it can be used like this:

squeue -u <username>

Interpreting Job state as provided by squeue

Jobs typically pass through several states in the course of their execution. Job state is shown in column "ST" of the output of squeue as an abbreviated code (e.g., "R" for RUNNING).

The most relevant codes and states are the following:

PD PENDING

Job is awaiting resource allocation.

R RUNNING

Job currently has an allocation.

S SUSPENDED

Job has an allocation, but execution has been suspended and CPUs have been released for other jobs.

CG COMPLETING

Job is in the process of completing. Some processes on some nodes may still be active.

CD COMPLETED

Job has terminated all processes on all nodes with an exit code of zero.

Beyond these, there are other (less frequent) job states. The SLURM doc page for squeue provides a complete list of them.

Knowing when jobs are expected to end or start

If you are interested in understanding when jobs are expected to start or end, use command

squeue -o "%5i %8u %10P %.2t |%19S |%.11L|"

which provides an output is similar to the following:

JOBID USER     PARTITION  ST |START_TIME          |  TIME_LEFT|
5307  thuynh   fat        PD |2022-11-11T17:55:54 | 3-00:00:00|
5308  thuynh   fat        PD |2022-11-11T17:55:54 | 3-00:00:00|
5296  cziyang  fat         R |2022-11-08T16:58:03 | 1-00:48:14|
5306  thuynh   fat         R |2022-11-10T08:13:30 | 2-16:03:41|
5297  gnannini fat         R |2022-11-08T17:55:54 | 1-01:46:05|
5336  ssaitta  gpu         R |2022-11-10T08:13:00 |    6:03:11|
5358  dmilesi  gpulong     R |2022-11-10T15:11:32 | 2-23:01:43|
5338  cziyang  gpulong     R |2022-11-10T09:45:01 | 1-17:35:12|

For running jobs (state R)

column "START_TIME" tells you when the job started its execution

column "TIME_LEFT" tells you how much remains of the running time requested by the job

For pending jobs (state PD)

column "START_TIME" tells you when the job is expected to start its execution

column "TIME_LEFT" tells you how much running time has been requested by the job

Important! Start and end times are forecasts based on the features of current jobs in the queues, and may change if running jobs end prematurely and/or if new jobs with higher priority are added to the queues. So these times should never be considered as certain.

If you simply want to know when pending jobs (state PD) are expected to begin execution, use

squeue --start

which lists pending jobs in order of increasing START_TIME (the job on top is the one which will be run first). For each pending job the command provides an output similar to the example below:

JOBID PARTITION     NAME     USER ST          START_TIME  NODES SCHEDNODES           NODELIST(REASON)
 5090       fat training   thuynh PD 2022-10-27T09:28:01      1 (null)               (Resources)

Getting detailed information about a job

If needed, complete information about a job (either pending or running) can be obtained using command

scontrol show job <JOBID>

where <JOBID> is the number from the first column of the output of squeue. The output of this command is similar to the following:

JobId=65 JobName=test_script.sh
   UserId=gfontana(10003) GroupId=gfontana(10004) MCS_label=N/A
   Priority=14208 Nice=0 Account=admin QOS=nogpu
   JobState=RUNNING Reason=None Dependency=(null)
   Requeue=0 Restarts=0 BatchFlag=0 Reboot=0 ExitCode=0:0
   RunTime=00:00:55 TimeLimit=01:00:00 TimeMin=N/A
   SubmitTime=2025-11-06T10:31:10 EligibleTime=2025-11-06T10:31:10
   AccrueTime=2025-11-06T10:31:10
   StartTime=2025-11-06T10:31:10 EndTime=2025-11-06T11:31:10 Deadline=N/A
   SuspendTime=None SecsPreSuspend=0 LastSchedEval=2025-11-06T10:31:10 Scheduler=Main
   Partition=jobs AllocNode:Sid=mufasa2-login:42020
   ReqNodeList=(null) ExcNodeList=(null)
   NodeList=gn01
   BatchHost=gn01
   NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
   ReqTRES=cpu=1,mem=4G,node=1,billing=1
   AllocTRES=cpu=1,mem=4G,node=1,billing=1
   Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
   MinCPUsNode=1 MinMemoryNode=4G MinTmpDiskNode=0
   Features=(null) DelayBoot=00:00:00
   OverSubscribe=OK Contiguous=0 Licenses=(null) LicensesAlloc=(null) Network=(null)
   Command=./test_script.sh
   WorkDir=/home/gfontana

In particular, the line beginning with "StartTime=" provides expected times for the start and end of job execution. As explained in Knowing when jobs are expected to end or start, start time is only a prediction and subject to change.

Cancelling a job with scancel

It is possible to cancel a job using command scancel, either while it is waiting for execution or when it is in execution (in this case you can choose what system signal to send the process in order to terminate it).

Please note that job priority for your user depends (also) on the overall duration of the jobs that you ran on Mufasa. Therefore, cancelling jobs that are not needed anymore improves your future jobs' priority.

The following are some examples of use of scancel adapted from SLURM's documentation.

scancel <JOBID>

removes queued job <JOBID> from the execution queue.

scancel --signal=TERM <JOBID>

terminates execution of job <JOBID> with signal SIGTERM (request to stop).

scancel --signal=KILL <JOBID>

terminates execution of job <JOBID> with signal SIGKILL (force stop).

scancel --state=PENDING --user=<username> --partition=<partition_name>

cancels all pending jobs belonging to user <username> in partition <partition_name>.

Knowing what jobs you ran today

Command

sacct -X

provides a list of all jobs run today by your user.