<?xml version="1.0"?>
<feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en">
	<id>https://biohpc.deib.polimi.it/api.php?action=feedcontributions&amp;feedformat=atom&amp;user=GiulioFontana</id>
	<title>Mufasa (BioHPC) - User contributions [en]</title>
	<link rel="self" type="application/atom+xml" href="https://biohpc.deib.polimi.it/api.php?action=feedcontributions&amp;feedformat=atom&amp;user=GiulioFontana"/>
	<link rel="alternate" type="text/html" href="https://biohpc.deib.polimi.it/index.php?title=Special:Contributions/GiulioFontana"/>
	<updated>2026-11-20T19:09:19Z</updated>
	<subtitle>User contributions</subtitle>
	<generator>MediaWiki 1.36.2</generator>
	<entry>
		<id>https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2607</id>
		<title>SLURM</title>
		<link rel="alternate" type="text/html" href="https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2607"/>
		<updated>2026-11-19T09:04:01Z</updated>

		<summary type="html">&lt;p&gt;GiulioFontana: /* SLURM in a nutshell */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page presents the features of SLURM that are most relevant to Mufasa&amp;#039;s [[Roles|Job Users]]. Job Users can submit jobs for execution, cancel their own jobs, and see other users&amp;#039; jobs (but not intervene on them).&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa &amp;#039;&amp;#039;&amp;#039;must use SLURM&amp;#039;&amp;#039;&amp;#039; to run resource-heavy processes, i.e. computing jobs that require one or more of the following:&lt;br /&gt;
* GPUs&lt;br /&gt;
* multiple CPUs&lt;br /&gt;
* powerful CPUs&lt;br /&gt;
* a significant amount of RAM&lt;br /&gt;
&lt;br /&gt;
In fact, only processes run via SLURM have access to all the resources of Mufasa. Processes run outside SLURM are executed by the [[System#Login server|login server]] virtual machine, which has minimal resources and no GPUs. Using SLURM is therefore the only way to execute resource-heavy jobs on Mufasa (this is a key difference between Mufasa 1.0 and Mufasa 2.0).&lt;br /&gt;
&lt;br /&gt;
= SLURM in a nutshell =&lt;br /&gt;
&lt;br /&gt;
Computation jobs on Mufasa needs to be launched via [[System#The SLURM job scheduling system|SLURM]]. SLURM provides jobs with access to the [[#System resources subjected to limitations|physical resources]] of Mufasa, such as CPUs, GPUs and RAM. Thanks to SLURM, processing jobs share system resources, optimising their occupation and availability. &lt;br /&gt;
&lt;br /&gt;
When a user runs a job, the job does not get executed immediately and is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039;. SLURM executes jobs according to their order in the queue: the top job in the queue gets executed as soon as the necessary resources are available, while jobs lower in the queue wait longer. The position of a job in the queue is due to the &amp;#039;&amp;#039;&amp;#039;priority&amp;#039;&amp;#039;&amp;#039; assigned to it by SLURM, with higher-priority jobs closer to the top. As a general rule:&lt;br /&gt;
&lt;br /&gt;
;: &amp;#039;&amp;#039;&amp;#039;the greater the fraction of Mufasa&amp;#039;s overall resources that a job asks for, the lower the job&amp;#039;s priority will be&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
The [[#Job priority|priority mechanism]] is used to encourage users to use Mufasa&amp;#039;s resources (i.e.: GPUs, CPUs, RAM, execution time) in an effective and equitable manner. This page includes a [[#How_to_maximise_the_priority_of_your_jobs|chart explaining how to maximise the priority of your jobs]].&lt;br /&gt;
&lt;br /&gt;
The &amp;#039;&amp;#039;&amp;#039;time&amp;#039;&amp;#039;&amp;#039; available to a job for its execution is controlled by SLURM. When a user requests execution of a job, they must specify the duration of the time slot that the job needs. The job must complete its execution before the end of the requested time slot, because at the end of such slot it gets killed by SLURM.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0 access to system resources is managed via SLURM&amp;#039;s [[#SLURM Quality of Service (QOS)|Quality of Service (QOS)]] mechanism (Mufasa 1.0 used [[#SLURM_partitions|partitions]] instead). While [[User Jobs#Running jobs with SLURM|launching a processing job via SLURM]], the user must always specify the chosen QOS. QOSes differ in the set of resources that they provide access to because each of them is designed to fit a given type of job.&lt;br /&gt;
&lt;br /&gt;
Mufasa sets limits to the number of jobs by the same user. This page includes a [[#Limits on jobs by the same user|table summarising such limits]].&lt;br /&gt;
&lt;br /&gt;
= SLURM Quality of Service (QOS) =&lt;br /&gt;
&lt;br /&gt;
Through &amp;#039;&amp;#039;&amp;#039;Quality of Services&amp;#039;&amp;#039;&amp;#039; (&amp;#039;&amp;#039;&amp;#039;QOSes&amp;#039;&amp;#039;&amp;#039;), SLURM lets system configurators assign a name to a set of related constraints.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, QOSes are used to define different levels of access to the server&amp;#039;s resources. When [[User Jobs|executing a job with SLURM]], a user must always &amp;#039;&amp;#039;&amp;#039;specify the QOS&amp;#039;&amp;#039;&amp;#039; that their job will use: this choice, in turn, determines what resources the job is able to access and influences the [[#Job priority|priority]] of the job.&lt;br /&gt;
&lt;br /&gt;
Mufasa&amp;#039;s QOSes and their features can be inspected with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list qos format=name%-11,priority,MaxSubmit,maxwall,maxtres%-80,mintres%-18&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following (see below for the same data in table form):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
Name          Priority MaxSubmit     MaxWall MaxTRES                                                                          MinTRES            &lt;br /&gt;
----------- ---------- --------- ----------- -------------------------------------------------------------------------------- ------------------ &lt;br /&gt;
normal               0                                                                                                                           &lt;br /&gt;
nogpu                4         1  3-00:00:00 cpu=16,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=128G                    &lt;br /&gt;
gpuheavy-20          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=2,mem=128G             gres/gpu:4g.20gb=1 &lt;br /&gt;
gpuheavy-40          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=1,gres/gpu:4g.20gb=0,mem=128G             gres/gpu:40gb=1    &lt;br /&gt;
gpulight             8         1    12:00:00 cpu=2,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpu                  2         1  1-00:00:00 cpu=8,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpuwide              2         2  1-00:00:00 cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=1,mem=64G              gres/gpu:4g.20gb=1 &lt;br /&gt;
build               32         1    02:00:00 cpu=2,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=16G&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The columns of this output are the following:&lt;br /&gt;
&lt;br /&gt;
:; Name&lt;br /&gt;
:: name of the QOS&lt;br /&gt;
&lt;br /&gt;
:; Priority&lt;br /&gt;
:: priority tier associated to the QOS (higher value = higher priority): see [[#Job priority|Job priority]] for details&lt;br /&gt;
&lt;br /&gt;
:; MaxSubmit&lt;br /&gt;
:: maximum number of jobs from a single user that can be submitted to SLURM with this QOS; submitted jobs include both running and queued jobs&lt;br /&gt;
:: See [[#Limits on jobs by the same user|Limits on jobs by the same user]] for an overview of the limits on jobs set by Mufasa.&lt;br /&gt;
&lt;br /&gt;
:; MaxWall&lt;br /&gt;
:: maximum wall clock duration of the jobs using the QOS (after which they are killed by SLURM), in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
:: For some QOSes these are not set: it means that they are determined by the [[#SLURM partitions|partition]]. Partitions also define the [[#Default values|default duration]] of jobs.&lt;br /&gt;
&lt;br /&gt;
:; MaxTRES&lt;br /&gt;
:: amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) available to a job using the QOS, where&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;cpu=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of CPUs (i.e., processor cores) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of CPUs specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;gres/&amp;#039;&amp;#039;gpu:Type&amp;#039;&amp;#039;=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of GPUs of class &amp;lt;code&amp;gt;&amp;#039;&amp;#039;Type&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; (see [[User Jobs#gres syntax|&amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax]]) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; (for QOSes that allow access to GPUs) if not specified, the job cannot be launched&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;mem=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;G&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum amount of system RAM is &amp;#039;&amp;#039;K&amp;#039;&amp;#039; GBytes&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of RAM specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
&lt;br /&gt;
:; MinTRES&lt;br /&gt;
:: minimum amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) that a job using the QOS must request in order to actually get executed by SLURM.&lt;br /&gt;
:: (If your job does not actually need these resources, you&amp;#039;ve chosen the wrong QOS: you can use one with a higher priority.)&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; QOS is the default one, and exists only to ensure that users always specify a QOS when running a job. Since &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; has zero priority and no resources, a job run using this QOS would never be run.&lt;br /&gt;
&lt;br /&gt;
The information provided by the &amp;lt;code&amp;gt;sacctmgr list qos&amp;lt;/code&amp;gt; command above is summarised by the following table:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! QOS&lt;br /&gt;
! Priority&amp;lt;/br&amp;gt;tier&lt;br /&gt;
! Max&amp;lt;/br&amp;gt;Submit&lt;br /&gt;
! MaxWall&amp;lt;/br&amp;gt;[h]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;CPUs&lt;br /&gt;
! max RAM&amp;lt;/br&amp;gt;[GB]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;3g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;4g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;40GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | build&lt;br /&gt;
| 32&lt;br /&gt;
| 1&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 16&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpulight&lt;br /&gt;
| 8&lt;br /&gt;
| 1&lt;br /&gt;
| 12&lt;br /&gt;
| 2&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | nogpu&lt;br /&gt;
| 4&lt;br /&gt;
| 1&lt;br /&gt;
| 72&lt;br /&gt;
| 16&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpu&lt;br /&gt;
| 2&lt;br /&gt;
| 1&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuwide&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-20&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| 2 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-40&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
A key piece of information in the table above is the &amp;#039;&amp;#039;&amp;#039;priority tier&amp;#039;&amp;#039;&amp;#039; associated to each QOS. In Mufasa 2.0, priority tiers are used to encourage users to use the &amp;#039;&amp;#039;&amp;#039;least powerful QOS that is compatible with their needs&amp;#039;&amp;#039;&amp;#039;, where &amp;quot;powerful&amp;quot; means &amp;quot;rich with resources&amp;quot;. Less powerful QOSes increase the priority of the jobs that use them, so these jobs tend to be executed sooner. See [[#Job priority|Job priority]] for details about how priority affects the execution order of jobs in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
== The &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS ==&lt;br /&gt;
&lt;br /&gt;
This QOS is specifically designed to be used by Mufasa users to &amp;#039;&amp;#039;&amp;#039;quickly build [[System#Containers|container images]]&amp;#039;&amp;#039;&amp;#039;. Its associated priority tier is very high: this, combined with the fact that jobs using this QOS require few resources, means that such jobs usually get executed very soon.&lt;br /&gt;
&lt;br /&gt;
On the other hand, the limited resources and the lack of access to the GPUs make the &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS unsuitable for other tasks.&lt;br /&gt;
&lt;br /&gt;
See [[Singularity#Building Singularity images|Building Singularity images]] for directions about building Singularity container images.&lt;br /&gt;
&lt;br /&gt;
== Restricted QOSes ==&lt;br /&gt;
&lt;br /&gt;
In Mufasa, the most powerful QOSes are reserved to researchers (called &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users: these include academic personnel and Ph.D. students). M.Sc. students, called &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users, cannot use them while running jobs.&lt;br /&gt;
&lt;br /&gt;
See [[#research users and students users|below]] to understand how &amp;#039;&amp;#039;user categories&amp;#039;&amp;#039; work in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
= &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users =&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa belong to two &amp;#039;&amp;#039;&amp;#039;user categories&amp;#039;&amp;#039;&amp;#039;. Categories provide users with different access to  to Mufasa&amp;#039;s resources: the idea being to provide researchers with more access while still letting students benefit from the server&amp;#039;s capabilities.&lt;br /&gt;
&lt;br /&gt;
User categories in Mufasa are the following:&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. academic personnel and Ph.D. students&lt;br /&gt;
::: * can use all QOSes, including the the [[#Restricted QOSes|restricted ones]]&lt;br /&gt;
::: * their jobs have a higher &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a higher number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. M.Sc. students&lt;br /&gt;
::: * cannot use the [[#Restricted QOSes|restricted QOSes]]&lt;br /&gt;
::: * their jobs have a lower &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a lower number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
You can inspect the differences in priority and running jobs between &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association format=&amp;quot;account,priority,maxjobs&amp;quot; | grep -E &amp;#039;Account|research|students&amp;#039;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
   Account   Priority MaxJobs &lt;br /&gt;
  research          4       2 &lt;br /&gt;
  students          1       1&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
To know what limits apply to your own user, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association where user=$USER format=&amp;quot;user,priority,maxjobs,qos%-60&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
      User   Priority MaxJobs QOS                                                          &lt;br /&gt;
---------- ---------- ------- ------------------------------------------------------------ &lt;br /&gt;
    preali          4       2 build,gpu,gpuheavy-20,gpuheavy-40,gpulight,gpuwide,nogpu&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The list under &amp;quot;QOS&amp;quot; shows what QOSes your user is allowed to use when [[User Jobs|running jobs]]. &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users can use all of them, while &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users can only access a subset of them.&lt;br /&gt;
&lt;br /&gt;
= Limits on jobs by the same user =&lt;br /&gt;
&lt;br /&gt;
Mufasa enforces limits on the number of jobs from a single user. Such limits aim at preventing users from &amp;quot;hogging&amp;quot; system resources, and apply to:&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;submitted jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that the user asked SLURM to execute, each of which may currently be either running or queued&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;running jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that are currently in execution&lt;br /&gt;
&lt;br /&gt;
Limits on submitted jobs are set via [[#SLURM Quality of Service (QOS)|QOSes]], while limits on running jobs are set via [[#research users and students users|user category]].&lt;br /&gt;
&lt;br /&gt;
The following table summarises what limits exist:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! &lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;running&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;submitted&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | global limits&amp;lt;br/&amp;gt;(system-wide)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to limits set by individual QOSes (below)&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | limits for&amp;lt;br/&amp;gt;individual&amp;lt;/br&amp;gt;QOSes&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to global limits (above)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for the&amp;#039;&amp;#039; &amp;lt;code&amp;gt;gpuwide&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;QOS&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for each of the other QOSes&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
= Job priority =&lt;br /&gt;
&lt;br /&gt;
Once the execution of a job has been requested, the job is not run immediately: it is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039; by SLURM, together with all the other jobs awaiting execution. The job on top of the queue at any time is the first to be put into execution as soon as the resources it requires are available. &lt;br /&gt;
&lt;br /&gt;
The order of the items in the job queue depends on their &amp;#039;&amp;#039;&amp;#039;priority&amp;#039;&amp;#039;&amp;#039;. The job with the highest priority is put by SLURM on top of the queue, while all other jobs are put in the queue in order of descending priority.&lt;br /&gt;
&lt;br /&gt;
The goal of SLURM is to maximise resource availability: i.e., to ensure the shorter possible wait time before job execution. To achieve this goal, in Mufasa SLURM is configured to &amp;#039;&amp;#039;&amp;#039;encourage users not to ask for resources or execution time that their job doesn&amp;#039;t need&amp;#039;&amp;#039;&amp;#039;. This is done via the priority mechanism: the more resources and/or time a job requests, the lower its priority will be; and the later it will be executed.&lt;br /&gt;
&lt;br /&gt;
Priority management in Mufasa is designed to set up a &amp;#039;&amp;#039;&amp;#039;virtuous cycle&amp;#039;&amp;#039;&amp;#039; where users, by carefully choosing what to ask for, obtain two results:&lt;br /&gt;
* they ensure that their job is executed as soon as possible;&lt;br /&gt;
* they leave as much as possible of Mufasa&amp;#039;s resources free for other users&amp;#039;s jobs.&lt;br /&gt;
&lt;br /&gt;
[[#How to maximise the priority of your jobs|Useful tips]] to maximise the priority of your jobs are provided below.&lt;br /&gt;
&lt;br /&gt;
== Elements determining job priority ==&lt;br /&gt;
In Mufasa, the priority of a job is computed by SLURM according to the following elements:&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#research users and students users|User category]]&amp;#039;&amp;#039;&amp;#039; (i.e., &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;)&lt;br /&gt;
::: Used to provide higher priority to jobs run by &amp;#039;&amp;#039;&amp;#039;research personnel&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#SLURM Quality of Service (QOS)|QOS]]&amp;#039;&amp;#039;&amp;#039; used by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;less resources&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Number of CPUs&amp;#039;&amp;#039;&amp;#039; requested by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;a lower number of CPUs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job Age&amp;#039;&amp;#039;&amp;#039;, i.e. the time that the job has been waiting in the queue&lt;br /&gt;
::: Used to provide higher priority to jobs which have been &amp;#039;&amp;#039;&amp;#039;queued for a longer time&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
::: (Periods when a queued job is not eligible for running because of the [[#Limits on jobs by the same user|limits on running jobs by the same user]] do not increase job age)&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;FairShare&amp;#039;&amp;#039;&amp;#039;, i.e. a factor computed by SLURM to balance use of the system by different users&lt;br /&gt;
::: Used to provide higher priority to jobs by users who &amp;#039;&amp;#039;&amp;#039;used Mufasa&amp;#039;s resources less than others&amp;#039;&amp;#039;&amp;#039; (see [[#How FairShare works|below]])&lt;br /&gt;
&lt;br /&gt;
You can understand how each of these elements contributes to the priority of the jobs waiting for execution (both yours and of other users) with command &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;. A tailored version of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; that only displays the parameters listed above is the following:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sprio -o &amp;quot;%6i %9u %.8Y %.10B %.5Q %.9T %.5A %.5F&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Here is an example of the output of the command above:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
JOBID  USER      PRIORITY      ASSOC   QOS      TRES   AGE FAIRS&lt;br /&gt;
5790   amoglia       3495       2500   375  cpu=-800   933   487&lt;br /&gt;
5791   qliu          3010       2500   375  cpu=-800   916    19&lt;br /&gt;
5817   mzoccola      2336        625   750  cpu=-800   495  1266&lt;br /&gt;
5822   fbottazzi     2235        625   750  cpu=-600   467   994&lt;br /&gt;
5837   lpassoni      2508       2500   750  cpu=-800     0    58&lt;br /&gt;
5838   bagostini     4437        625  3000  cpu=-200     0  1013&lt;br /&gt;
5839   flazzaro      1549        625   750  cpu=-800     0   974&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
In it, column PRIORITY provides the overall priority of each job. Each priority value is the sum of the five values to its right; in particular, column ASSOC corresponds to user category, while column TRES corresponds to the number of CPUs. Note that the contribution of TRES to priority is &amp;#039;&amp;#039;negative&amp;#039;&amp;#039;, i.e. it is a penalty; its value is proportional to the number of CPUs requested by the job.&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;Things to consider when using &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
* Its output is only relevant for jobs &amp;#039;&amp;#039;not yet in execution&amp;#039;&amp;#039;. For an already running job, the values shown by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; correspond to the priority that the job would get if evaluated now: however, this priority does not affect job execution in any way.&lt;br /&gt;
* Jobs listed by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; may temporarily differ from jobs listed by &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;: this is due to the fact that SLURM updates its internal data (used by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;) every few minutes, while the output of &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt; is updated much more frequently.&lt;br /&gt;
&lt;br /&gt;
=== How FairShare works ===&lt;br /&gt;
In Mufasa 2.0, the FairShare of each user is computed according to these rules:&lt;br /&gt;
# It considers the quantity used by (current and past) user jobs of:&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;CPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of CPUs used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;RAM&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the amount of RAM used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;GPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of GPUs used&lt;br /&gt;
#** note: 40 GB GPUs have double the impact on FairShare than 20 GB GPUs&lt;br /&gt;
# The impact on FairShare of any use of resources is &amp;#039;&amp;#039;&amp;#039;proportional to its duration&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
#* example: using 32 GB of RAM for 48h has the same impact of using 64 GB of RAM for 24h&lt;br /&gt;
# FairShare has a &amp;quot;fading memory&amp;quot;: i.e., resource use has less and less impact on FairShare the farther in the past it is&lt;br /&gt;
#* consequence: over time, the &amp;quot;history&amp;quot; of a user gets forgotten by FairShare&lt;br /&gt;
&lt;br /&gt;
== How to maximise the priority of your jobs ==&lt;br /&gt;
&lt;br /&gt;
Every time you run a SLURM job, follow these guidelines:&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Choose the less powerful QOS compatible with the needs of your job&lt;br /&gt;
:: QOSes with access to less resources have higher base priority.&lt;br /&gt;
&lt;br /&gt;
; Only request CPUs and RAM that your job really needs&lt;br /&gt;
:: Usually, code exploits multiple CPUs only if specifically designed to do so: if unsure, only ask for 1 CPU.&lt;br /&gt;
:: The fewer resources your job asks for, the less it will wait before they become available.&lt;br /&gt;
:: The more CPUs your job asks for, the stronger the penalty to its priority.&lt;br /&gt;
:: Asking for fewer CPUs and/or less RAM improves your FairShare.&lt;br /&gt;
&lt;br /&gt;
; Do not request more time than your jobs needs to complete&lt;br /&gt;
:: Make a worst-case estimate and only ask for that duration.&lt;br /&gt;
:: Asking for less of Mufasa&amp;#039;s time improves your FairShare.&lt;br /&gt;
&lt;br /&gt;
; Debug and test code using less powerful QOSes before running it with more powerful QOSes&lt;br /&gt;
:: Your test jobs will get a higher priority and your FairShare will improve.&lt;br /&gt;
&lt;br /&gt;
; Cancel jobs when you don&amp;#039;t need them anymore&lt;br /&gt;
:: Always use [[User_Jobs#Cancelling_a_job_with_scancel|scancel]] to delete completed (or crashed) jobs: your Fairshare will improve.&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Protip: it&amp;#039;s a good idea to [[User Jobs#Looking for unused GPUs|check for unused GPUs]] before choosing what to request. Requesting a GPU that is currently idle will help your job get executed sooner.&lt;br /&gt;
&lt;br /&gt;
= SLURM partitions =&lt;br /&gt;
&lt;br /&gt;
Partitions are another mechanism provided by SLURM to create different levels of access to system resources. Since in Mufasa 2.0 access to resources is controlled via [[#SLURM Quality of Service (QOS)|QOSes]], partitions are not very relevant. (This is one of the main differences between Mufasa 1.0 and Mufasa 2.0.)&lt;br /&gt;
&lt;br /&gt;
Note, however, that the default values for some features of SLURM jobs (e.g., duration) are [[#Default values|set by the partition]].&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, there is a single SLURM partition, called &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt;, and all jobs run on it. The state of &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt; can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sinfo -o &amp;quot;%10P %5a %9T %11L %10l&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PARTITION  AVAIL STATE     DEFAULTTIME TIMELIMIT &lt;br /&gt;
jobs*      up    idle      1:00:00     3-00:00:00&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where columns correspond to the following information:&lt;br /&gt;
&lt;br /&gt;
:; PARTITION&lt;br /&gt;
:: name of the partition; the asterisks indicates that it&amp;#039;s the default one&lt;br /&gt;
&lt;br /&gt;
:; AVAIL&lt;br /&gt;
:: state/availability of the partition: see [[#Partition availability|below]]&lt;br /&gt;
&lt;br /&gt;
:; STATE&lt;br /&gt;
:: state (using [https://slurm.schedmd.com/sinfo.html#SECTION_NODE-STATE-CODES these codes])&lt;br /&gt;
:: typical values are &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;mixed&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that some of the resources are busy executing jobs while other are idle, and &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;allocated&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that all of the resources are in use&lt;br /&gt;
&lt;br /&gt;
:; DEFAULTTIME&lt;br /&gt;
:: default runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:; TIMELIMIT&lt;br /&gt;
:: maximum runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
The asterisk at the end of the partition name indicates the default partition, i.e. the one on which jobs which do not ask for a specific partition are run.&lt;br /&gt;
&lt;br /&gt;
Command &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; does not tell you about the &amp;#039;&amp;#039;jobs&amp;#039;&amp;#039; submitted to a partition. This information is obtained, instead, with [[User Jobs#Inspecting jobs with squeue|command &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;]].&lt;br /&gt;
&lt;br /&gt;
== Partition availability ==&lt;br /&gt;
&lt;br /&gt;
The most important information that &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; provides is the &amp;#039;&amp;#039;&amp;#039;availability&amp;#039;&amp;#039;&amp;#039; (also called &amp;#039;&amp;#039;&amp;#039;state&amp;#039;&amp;#039;&amp;#039;) of partitions. This is shown in column &amp;quot;AVAIL&amp;quot;. Possible partition states are:&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is available&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed as soon as resources allow&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is in the process of becoming unavailable (i.e., of entering the &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; state: see below)&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is unavailable&lt;br /&gt;
:: There are no running jobs&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
When a partition goes from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; no harm is done to running jobs. In a normally functioning SLURM system, the passage from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; happens only when no jobs are running on the partition. If (e.g., due to a malfunction) the passage happens with jobs still running, they get killed.&lt;br /&gt;
&lt;br /&gt;
A partition in state &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; requires intervention by a [[Roles|Job Administrator]] to be restored to &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
== Default values ==&lt;br /&gt;
&lt;br /&gt;
The features of SLURM partitions, including the &amp;#039;&amp;#039;&amp;#039;default values&amp;#039;&amp;#039;&amp;#039; which are applied to jobs that do not make explicit requests, can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scontrol show partition&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PartitionName=jobs&lt;br /&gt;
   AllowGroups=ALL AllowAccounts=ALL AllowQos=nogpu,gpulight,gpu,gpuwide,gpuheavy-20,gpuheavy-40&lt;br /&gt;
   AllocNodes=ALL Default=YES QoS=N/A&lt;br /&gt;
   DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO&lt;br /&gt;
   MaxNodes=UNLIMITED MaxTime=3-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED&lt;br /&gt;
   Nodes=gn01&lt;br /&gt;
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO&lt;br /&gt;
   OverTimeLimit=NONE PreemptMode=OFF&lt;br /&gt;
   State=UP TotalCPUs=48 TotalNodes=1 SelectTypeParameters=NONE&lt;br /&gt;
   JobDefaults=(null)&lt;br /&gt;
   DefMemPerNode=4096 MaxMemPerNode=UNLIMITED&lt;br /&gt;
   TRES=cpu=48,mem=1011435M,node=1,billing=49,gres/gpu=13,gres/gpu:3g.20gb=5,gres/gpu:40gb=3,gres/gpu:4g.20gb=5&lt;br /&gt;
   TRESBillingWeights=cpu=1.0,gres/gpu:3g.20gb=6.0,gres/gpu:4g.20gb=6.0,gres/gpu:40gb=6.0,mem=0.05g&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In the output of &amp;lt;code&amp;gt;scontrol show partition&amp;lt;/code&amp;gt; the most relevant default values for Mufasa users are:&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefaultTime&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default execution time assigned to a job run on the partition (e.g., 1 hour)&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefMemPerNode&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default amount of RAM assigned to a job run on the partition (e.g., 4GB)&lt;br /&gt;
&lt;br /&gt;
= System resources subjected to limitations =&lt;br /&gt;
&lt;br /&gt;
In systems based on SLURM like Mufasa, &amp;#039;&amp;#039;&amp;#039;TRES (Trackable RESources)&amp;#039;&amp;#039;&amp;#039; are (from [https://slurm.schedmd.com/tres.html SLURM&amp;#039;s documentation] &amp;quot;&amp;#039;&amp;#039;resources that can be tracked for usage or used to enforce limits against.&amp;#039;&amp;#039;&amp;quot;&lt;br /&gt;
&lt;br /&gt;
TRES include CPUs, RAM and &amp;#039;&amp;#039;&amp;#039;GRES&amp;#039;&amp;#039;&amp;#039;. The last term stands for &amp;#039;&amp;#039;Generic RESources&amp;#039;&amp;#039; that a job may need for its execution. In Mufasa, the only &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; resources are the GPUs.&lt;br /&gt;
&lt;br /&gt;
== &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax ==&lt;br /&gt;
&lt;br /&gt;
To ask SLURM to assign GRES resources (i.e., GPUs) to a job, a special syntax must be used. Precisely, the name of each GPU resource takes the form&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:Name:Type&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
Considering the [[System#CPUs and GPUs|GPU complement of Mufasa]], &amp;lt;code&amp;gt;Type&amp;lt;/code&amp;gt; takes the following values:&lt;br /&gt;
&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:40gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 40 Gbytes of RAM&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:4g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 4 compute units&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 3 compute units&lt;br /&gt;
&lt;br /&gt;
So, for instance,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
identifies a resource corresponding to a GPU with 20 GB of RAM and 3 compute units.&lt;br /&gt;
&lt;br /&gt;
When asking for a GRES resource (e.g., in an &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command or an &amp;lt;code&amp;gt;SBATCH&amp;lt;/code&amp;gt; directive of an [[User Jobs#Using execution scripts to run jobs|execution script]]), the syntax required by SLURM is&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:&amp;lt;Type&amp;gt;:&amp;lt;Quantity&amp;gt;&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
where &amp;lt;code&amp;gt;Quantity&amp;lt;/code&amp;gt; is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of type &amp;lt;code&amp;gt;4g.20gb&amp;lt;/code&amp;gt; the syntax is&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:4g.20gb:2&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SLURM&amp;#039;s &amp;#039;&amp;#039;generic resources&amp;#039;&amp;#039; are defined in &amp;lt;code&amp;gt;/etc/slurm/gres.conf&amp;lt;/code&amp;gt;. In order to make GPUs available to SLURM&amp;#039;s &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; management, Mufasa makes use of Nvidia&amp;#039;s [https://developer.nvidia.com/nvidia-management-library-nvml NVML library]. For additional information see [https://slurm.schedmd.com/gres.html SLURM&amp;#039;s documentation].&lt;/div&gt;</summary>
		<author><name>GiulioFontana</name></author>
	</entry>
	<entry>
		<id>https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2606</id>
		<title>SLURM</title>
		<link rel="alternate" type="text/html" href="https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2606"/>
		<updated>2026-11-19T09:00:16Z</updated>

		<summary type="html">&lt;p&gt;GiulioFontana: /* SLURM in a nutshell */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page presents the features of SLURM that are most relevant to Mufasa&amp;#039;s [[Roles|Job Users]]. Job Users can submit jobs for execution, cancel their own jobs, and see other users&amp;#039; jobs (but not intervene on them).&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa &amp;#039;&amp;#039;&amp;#039;must use SLURM&amp;#039;&amp;#039;&amp;#039; to run resource-heavy processes, i.e. computing jobs that require one or more of the following:&lt;br /&gt;
* GPUs&lt;br /&gt;
* multiple CPUs&lt;br /&gt;
* powerful CPUs&lt;br /&gt;
* a significant amount of RAM&lt;br /&gt;
&lt;br /&gt;
In fact, only processes run via SLURM have access to all the resources of Mufasa. Processes run outside SLURM are executed by the [[System#Login server|login server]] virtual machine, which has minimal resources and no GPUs. Using SLURM is therefore the only way to execute resource-heavy jobs on Mufasa (this is a key difference between Mufasa 1.0 and Mufasa 2.0).&lt;br /&gt;
&lt;br /&gt;
= SLURM in a nutshell =&lt;br /&gt;
&lt;br /&gt;
Computation jobs on Mufasa needs to be launched via [[System#The SLURM job scheduling system|SLURM]]. SLURM provides jobs with access to the [[#System resources subjected to limitations|physical resources]] of Mufasa, such as CPUs, GPUs and RAM. Thanks to SLURM, processing jobs share system resources, optimising their occupation and availability. &lt;br /&gt;
&lt;br /&gt;
When a user runs a job, the job does not get executed immediately and is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039;. SLURM executes jobs according to their order in the queue: the top job in the queue gets executed as soon as the necessary resources are available, while jobs lower in the queue wait longer. The position of a job in the queue is due to the [[#Job priority|priority]] assigned to it by SLURM, with higher-priority jobs closer to the top. As a general rule:&lt;br /&gt;
&lt;br /&gt;
;: &amp;#039;&amp;#039;&amp;#039;the greater the fraction of Mufasa&amp;#039;s overall resources that a job asks for, the lower the job&amp;#039;s priority will be&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
The [[#Job priority|priority mechanism]] is used to encourage users to use Mufasa&amp;#039;s resources (i.e.: GPUs, CPUs, RAM, execution time) in an effective and equitable manner. This page includes a [[#How_to_maximise_the_priority_of_your_jobs|chart explaining how to maximise the priority of your jobs]].&lt;br /&gt;
&lt;br /&gt;
The &amp;#039;&amp;#039;&amp;#039;time&amp;#039;&amp;#039;&amp;#039; available to a job for its execution is controlled by SLURM. When a user requests execution of a job, they must specify the duration of the time slot that the job needs. The job must complete its execution before the end of the requested time slot, because at the end of such slot it gets killed by SLURM.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0 access to system resources is managed via SLURM&amp;#039;s [[#SLURM Quality of Service (QOS)|Quality of Service (QOS)]] mechanism (Mufasa 1.0 used [[#SLURM_partitions|partitions]] instead). While [[User Jobs#Running jobs with SLURM|launching a processing job via SLURM]], the user must always specify the chosen QOS. QOSes differ in the set of resources that they provide access to because each of them is designed to fit a given type of job.&lt;br /&gt;
&lt;br /&gt;
Mufasa sets limits to the number of jobs by the same user. This page includes a [[#Limits on jobs by the same user|table summarising such limits]].&lt;br /&gt;
&lt;br /&gt;
= SLURM Quality of Service (QOS) =&lt;br /&gt;
&lt;br /&gt;
Through &amp;#039;&amp;#039;&amp;#039;Quality of Services&amp;#039;&amp;#039;&amp;#039; (&amp;#039;&amp;#039;&amp;#039;QOSes&amp;#039;&amp;#039;&amp;#039;), SLURM lets system configurators assign a name to a set of related constraints.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, QOSes are used to define different levels of access to the server&amp;#039;s resources. When [[User Jobs|executing a job with SLURM]], a user must always &amp;#039;&amp;#039;&amp;#039;specify the QOS&amp;#039;&amp;#039;&amp;#039; that their job will use: this choice, in turn, determines what resources the job is able to access and influences the [[#Job priority|priority]] of the job.&lt;br /&gt;
&lt;br /&gt;
Mufasa&amp;#039;s QOSes and their features can be inspected with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list qos format=name%-11,priority,MaxSubmit,maxwall,maxtres%-80,mintres%-18&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following (see below for the same data in table form):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
Name          Priority MaxSubmit     MaxWall MaxTRES                                                                          MinTRES            &lt;br /&gt;
----------- ---------- --------- ----------- -------------------------------------------------------------------------------- ------------------ &lt;br /&gt;
normal               0                                                                                                                           &lt;br /&gt;
nogpu                4         1  3-00:00:00 cpu=16,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=128G                    &lt;br /&gt;
gpuheavy-20          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=2,mem=128G             gres/gpu:4g.20gb=1 &lt;br /&gt;
gpuheavy-40          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=1,gres/gpu:4g.20gb=0,mem=128G             gres/gpu:40gb=1    &lt;br /&gt;
gpulight             8         1    12:00:00 cpu=2,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpu                  2         1  1-00:00:00 cpu=8,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpuwide              2         2  1-00:00:00 cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=1,mem=64G              gres/gpu:4g.20gb=1 &lt;br /&gt;
build               32         1    02:00:00 cpu=2,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=16G&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The columns of this output are the following:&lt;br /&gt;
&lt;br /&gt;
:; Name&lt;br /&gt;
:: name of the QOS&lt;br /&gt;
&lt;br /&gt;
:; Priority&lt;br /&gt;
:: priority tier associated to the QOS (higher value = higher priority): see [[#Job priority|Job priority]] for details&lt;br /&gt;
&lt;br /&gt;
:; MaxSubmit&lt;br /&gt;
:: maximum number of jobs from a single user that can be submitted to SLURM with this QOS; submitted jobs include both running and queued jobs&lt;br /&gt;
:: See [[#Limits on jobs by the same user|Limits on jobs by the same user]] for an overview of the limits on jobs set by Mufasa.&lt;br /&gt;
&lt;br /&gt;
:; MaxWall&lt;br /&gt;
:: maximum wall clock duration of the jobs using the QOS (after which they are killed by SLURM), in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
:: For some QOSes these are not set: it means that they are determined by the [[#SLURM partitions|partition]]. Partitions also define the [[#Default values|default duration]] of jobs.&lt;br /&gt;
&lt;br /&gt;
:; MaxTRES&lt;br /&gt;
:: amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) available to a job using the QOS, where&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;cpu=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of CPUs (i.e., processor cores) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of CPUs specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;gres/&amp;#039;&amp;#039;gpu:Type&amp;#039;&amp;#039;=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of GPUs of class &amp;lt;code&amp;gt;&amp;#039;&amp;#039;Type&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; (see [[User Jobs#gres syntax|&amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax]]) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; (for QOSes that allow access to GPUs) if not specified, the job cannot be launched&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;mem=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;G&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum amount of system RAM is &amp;#039;&amp;#039;K&amp;#039;&amp;#039; GBytes&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of RAM specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
&lt;br /&gt;
:; MinTRES&lt;br /&gt;
:: minimum amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) that a job using the QOS must request in order to actually get executed by SLURM.&lt;br /&gt;
:: (If your job does not actually need these resources, you&amp;#039;ve chosen the wrong QOS: you can use one with a higher priority.)&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; QOS is the default one, and exists only to ensure that users always specify a QOS when running a job. Since &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; has zero priority and no resources, a job run using this QOS would never be run.&lt;br /&gt;
&lt;br /&gt;
The information provided by the &amp;lt;code&amp;gt;sacctmgr list qos&amp;lt;/code&amp;gt; command above is summarised by the following table:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! QOS&lt;br /&gt;
! Priority&amp;lt;/br&amp;gt;tier&lt;br /&gt;
! Max&amp;lt;/br&amp;gt;Submit&lt;br /&gt;
! MaxWall&amp;lt;/br&amp;gt;[h]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;CPUs&lt;br /&gt;
! max RAM&amp;lt;/br&amp;gt;[GB]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;3g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;4g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;40GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | build&lt;br /&gt;
| 32&lt;br /&gt;
| 1&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 16&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpulight&lt;br /&gt;
| 8&lt;br /&gt;
| 1&lt;br /&gt;
| 12&lt;br /&gt;
| 2&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | nogpu&lt;br /&gt;
| 4&lt;br /&gt;
| 1&lt;br /&gt;
| 72&lt;br /&gt;
| 16&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpu&lt;br /&gt;
| 2&lt;br /&gt;
| 1&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuwide&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-20&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| 2 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-40&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
A key piece of information in the table above is the &amp;#039;&amp;#039;&amp;#039;priority tier&amp;#039;&amp;#039;&amp;#039; associated to each QOS. In Mufasa 2.0, priority tiers are used to encourage users to use the &amp;#039;&amp;#039;&amp;#039;least powerful QOS that is compatible with their needs&amp;#039;&amp;#039;&amp;#039;, where &amp;quot;powerful&amp;quot; means &amp;quot;rich with resources&amp;quot;. Less powerful QOSes increase the priority of the jobs that use them, so these jobs tend to be executed sooner. See [[#Job priority|Job priority]] for details about how priority affects the execution order of jobs in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
== The &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS ==&lt;br /&gt;
&lt;br /&gt;
This QOS is specifically designed to be used by Mufasa users to &amp;#039;&amp;#039;&amp;#039;quickly build [[System#Containers|container images]]&amp;#039;&amp;#039;&amp;#039;. Its associated priority tier is very high: this, combined with the fact that jobs using this QOS require few resources, means that such jobs usually get executed very soon.&lt;br /&gt;
&lt;br /&gt;
On the other hand, the limited resources and the lack of access to the GPUs make the &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS unsuitable for other tasks.&lt;br /&gt;
&lt;br /&gt;
See [[Singularity#Building Singularity images|Building Singularity images]] for directions about building Singularity container images.&lt;br /&gt;
&lt;br /&gt;
== Restricted QOSes ==&lt;br /&gt;
&lt;br /&gt;
In Mufasa, the most powerful QOSes are reserved to researchers (called &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users: these include academic personnel and Ph.D. students). M.Sc. students, called &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users, cannot use them while running jobs.&lt;br /&gt;
&lt;br /&gt;
See [[#research users and students users|below]] to understand how &amp;#039;&amp;#039;user categories&amp;#039;&amp;#039; work in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
= &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users =&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa belong to two &amp;#039;&amp;#039;&amp;#039;user categories&amp;#039;&amp;#039;&amp;#039;. Categories provide users with different access to  to Mufasa&amp;#039;s resources: the idea being to provide researchers with more access while still letting students benefit from the server&amp;#039;s capabilities.&lt;br /&gt;
&lt;br /&gt;
User categories in Mufasa are the following:&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. academic personnel and Ph.D. students&lt;br /&gt;
::: * can use all QOSes, including the the [[#Restricted QOSes|restricted ones]]&lt;br /&gt;
::: * their jobs have a higher &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a higher number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. M.Sc. students&lt;br /&gt;
::: * cannot use the [[#Restricted QOSes|restricted QOSes]]&lt;br /&gt;
::: * their jobs have a lower &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a lower number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
You can inspect the differences in priority and running jobs between &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association format=&amp;quot;account,priority,maxjobs&amp;quot; | grep -E &amp;#039;Account|research|students&amp;#039;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
   Account   Priority MaxJobs &lt;br /&gt;
  research          4       2 &lt;br /&gt;
  students          1       1&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
To know what limits apply to your own user, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association where user=$USER format=&amp;quot;user,priority,maxjobs,qos%-60&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
      User   Priority MaxJobs QOS                                                          &lt;br /&gt;
---------- ---------- ------- ------------------------------------------------------------ &lt;br /&gt;
    preali          4       2 build,gpu,gpuheavy-20,gpuheavy-40,gpulight,gpuwide,nogpu&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The list under &amp;quot;QOS&amp;quot; shows what QOSes your user is allowed to use when [[User Jobs|running jobs]]. &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users can use all of them, while &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users can only access a subset of them.&lt;br /&gt;
&lt;br /&gt;
= Limits on jobs by the same user =&lt;br /&gt;
&lt;br /&gt;
Mufasa enforces limits on the number of jobs from a single user. Such limits aim at preventing users from &amp;quot;hogging&amp;quot; system resources, and apply to:&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;submitted jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that the user asked SLURM to execute, each of which may currently be either running or queued&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;running jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that are currently in execution&lt;br /&gt;
&lt;br /&gt;
Limits on submitted jobs are set via [[#SLURM Quality of Service (QOS)|QOSes]], while limits on running jobs are set via [[#research users and students users|user category]].&lt;br /&gt;
&lt;br /&gt;
The following table summarises what limits exist:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! &lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;running&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;submitted&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | global limits&amp;lt;br/&amp;gt;(system-wide)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to limits set by individual QOSes (below)&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | limits for&amp;lt;br/&amp;gt;individual&amp;lt;/br&amp;gt;QOSes&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to global limits (above)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for the&amp;#039;&amp;#039; &amp;lt;code&amp;gt;gpuwide&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;QOS&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for each of the other QOSes&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
= Job priority =&lt;br /&gt;
&lt;br /&gt;
Once the execution of a job has been requested, the job is not run immediately: it is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039; by SLURM, together with all the other jobs awaiting execution. The job on top of the queue at any time is the first to be put into execution as soon as the resources it requires are available. &lt;br /&gt;
&lt;br /&gt;
The order of the items in the job queue depends on their &amp;#039;&amp;#039;&amp;#039;priority&amp;#039;&amp;#039;&amp;#039;. The job with the highest priority is put by SLURM on top of the queue, while all other jobs are put in the queue in order of descending priority.&lt;br /&gt;
&lt;br /&gt;
The goal of SLURM is to maximise resource availability: i.e., to ensure the shorter possible wait time before job execution. To achieve this goal, in Mufasa SLURM is configured to &amp;#039;&amp;#039;&amp;#039;encourage users not to ask for resources or execution time that their job doesn&amp;#039;t need&amp;#039;&amp;#039;&amp;#039;. This is done via the priority mechanism: the more resources and/or time a job requests, the lower its priority will be; and the later it will be executed.&lt;br /&gt;
&lt;br /&gt;
Priority management in Mufasa is designed to set up a &amp;#039;&amp;#039;&amp;#039;virtuous cycle&amp;#039;&amp;#039;&amp;#039; where users, by carefully choosing what to ask for, obtain two results:&lt;br /&gt;
* they ensure that their job is executed as soon as possible;&lt;br /&gt;
* they leave as much as possible of Mufasa&amp;#039;s resources free for other users&amp;#039;s jobs.&lt;br /&gt;
&lt;br /&gt;
[[#How to maximise the priority of your jobs|Useful tips]] to maximise the priority of your jobs are provided below.&lt;br /&gt;
&lt;br /&gt;
== Elements determining job priority ==&lt;br /&gt;
In Mufasa, the priority of a job is computed by SLURM according to the following elements:&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#research users and students users|User category]]&amp;#039;&amp;#039;&amp;#039; (i.e., &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;)&lt;br /&gt;
::: Used to provide higher priority to jobs run by &amp;#039;&amp;#039;&amp;#039;research personnel&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#SLURM Quality of Service (QOS)|QOS]]&amp;#039;&amp;#039;&amp;#039; used by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;less resources&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Number of CPUs&amp;#039;&amp;#039;&amp;#039; requested by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;a lower number of CPUs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job Age&amp;#039;&amp;#039;&amp;#039;, i.e. the time that the job has been waiting in the queue&lt;br /&gt;
::: Used to provide higher priority to jobs which have been &amp;#039;&amp;#039;&amp;#039;queued for a longer time&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
::: (Periods when a queued job is not eligible for running because of the [[#Limits on jobs by the same user|limits on running jobs by the same user]] do not increase job age)&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;FairShare&amp;#039;&amp;#039;&amp;#039;, i.e. a factor computed by SLURM to balance use of the system by different users&lt;br /&gt;
::: Used to provide higher priority to jobs by users who &amp;#039;&amp;#039;&amp;#039;used Mufasa&amp;#039;s resources less than others&amp;#039;&amp;#039;&amp;#039; (see [[#How FairShare works|below]])&lt;br /&gt;
&lt;br /&gt;
You can understand how each of these elements contributes to the priority of the jobs waiting for execution (both yours and of other users) with command &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;. A tailored version of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; that only displays the parameters listed above is the following:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sprio -o &amp;quot;%6i %9u %.8Y %.10B %.5Q %.9T %.5A %.5F&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Here is an example of the output of the command above:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
JOBID  USER      PRIORITY      ASSOC   QOS      TRES   AGE FAIRS&lt;br /&gt;
5790   amoglia       3495       2500   375  cpu=-800   933   487&lt;br /&gt;
5791   qliu          3010       2500   375  cpu=-800   916    19&lt;br /&gt;
5817   mzoccola      2336        625   750  cpu=-800   495  1266&lt;br /&gt;
5822   fbottazzi     2235        625   750  cpu=-600   467   994&lt;br /&gt;
5837   lpassoni      2508       2500   750  cpu=-800     0    58&lt;br /&gt;
5838   bagostini     4437        625  3000  cpu=-200     0  1013&lt;br /&gt;
5839   flazzaro      1549        625   750  cpu=-800     0   974&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
In it, column PRIORITY provides the overall priority of each job. Each priority value is the sum of the five values to its right; in particular, column ASSOC corresponds to user category, while column TRES corresponds to the number of CPUs. Note that the contribution of TRES to priority is &amp;#039;&amp;#039;negative&amp;#039;&amp;#039;, i.e. it is a penalty; its value is proportional to the number of CPUs requested by the job.&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;Things to consider when using &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
* Its output is only relevant for jobs &amp;#039;&amp;#039;not yet in execution&amp;#039;&amp;#039;. For an already running job, the values shown by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; correspond to the priority that the job would get if evaluated now: however, this priority does not affect job execution in any way.&lt;br /&gt;
* Jobs listed by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; may temporarily differ from jobs listed by &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;: this is due to the fact that SLURM updates its internal data (used by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;) every few minutes, while the output of &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt; is updated much more frequently.&lt;br /&gt;
&lt;br /&gt;
=== How FairShare works ===&lt;br /&gt;
In Mufasa 2.0, the FairShare of each user is computed according to these rules:&lt;br /&gt;
# It considers the quantity used by (current and past) user jobs of:&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;CPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of CPUs used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;RAM&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the amount of RAM used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;GPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of GPUs used&lt;br /&gt;
#** note: 40 GB GPUs have double the impact on FairShare than 20 GB GPUs&lt;br /&gt;
# The impact on FairShare of any use of resources is &amp;#039;&amp;#039;&amp;#039;proportional to its duration&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
#* example: using 32 GB of RAM for 48h has the same impact of using 64 GB of RAM for 24h&lt;br /&gt;
# FairShare has a &amp;quot;fading memory&amp;quot;: i.e., resource use has less and less impact on FairShare the farther in the past it is&lt;br /&gt;
#* consequence: over time, the &amp;quot;history&amp;quot; of a user gets forgotten by FairShare&lt;br /&gt;
&lt;br /&gt;
== How to maximise the priority of your jobs ==&lt;br /&gt;
&lt;br /&gt;
Every time you run a SLURM job, follow these guidelines:&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Choose the less powerful QOS compatible with the needs of your job&lt;br /&gt;
:: QOSes with access to less resources have higher base priority.&lt;br /&gt;
&lt;br /&gt;
; Only request CPUs and RAM that your job really needs&lt;br /&gt;
:: Usually, code exploits multiple CPUs only if specifically designed to do so: if unsure, only ask for 1 CPU.&lt;br /&gt;
:: The fewer resources your job asks for, the less it will wait before they become available.&lt;br /&gt;
:: The more CPUs your job asks for, the stronger the penalty to its priority.&lt;br /&gt;
:: Asking for fewer CPUs and/or less RAM improves your FairShare.&lt;br /&gt;
&lt;br /&gt;
; Do not request more time than your jobs needs to complete&lt;br /&gt;
:: Make a worst-case estimate and only ask for that duration.&lt;br /&gt;
:: Asking for less of Mufasa&amp;#039;s time improves your FairShare.&lt;br /&gt;
&lt;br /&gt;
; Debug and test code using less powerful QOSes before running it with more powerful QOSes&lt;br /&gt;
:: Your test jobs will get a higher priority and your FairShare will improve.&lt;br /&gt;
&lt;br /&gt;
; Cancel jobs when you don&amp;#039;t need them anymore&lt;br /&gt;
:: Always use [[User_Jobs#Cancelling_a_job_with_scancel|scancel]] to delete completed (or crashed) jobs: your Fairshare will improve.&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Protip: it&amp;#039;s a good idea to [[User Jobs#Looking for unused GPUs|check for unused GPUs]] before choosing what to request. Requesting a GPU that is currently idle will help your job get executed sooner.&lt;br /&gt;
&lt;br /&gt;
= SLURM partitions =&lt;br /&gt;
&lt;br /&gt;
Partitions are another mechanism provided by SLURM to create different levels of access to system resources. Since in Mufasa 2.0 access to resources is controlled via [[#SLURM Quality of Service (QOS)|QOSes]], partitions are not very relevant. (This is one of the main differences between Mufasa 1.0 and Mufasa 2.0.)&lt;br /&gt;
&lt;br /&gt;
Note, however, that the default values for some features of SLURM jobs (e.g., duration) are [[#Default values|set by the partition]].&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, there is a single SLURM partition, called &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt;, and all jobs run on it. The state of &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt; can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sinfo -o &amp;quot;%10P %5a %9T %11L %10l&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PARTITION  AVAIL STATE     DEFAULTTIME TIMELIMIT &lt;br /&gt;
jobs*      up    idle      1:00:00     3-00:00:00&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where columns correspond to the following information:&lt;br /&gt;
&lt;br /&gt;
:; PARTITION&lt;br /&gt;
:: name of the partition; the asterisks indicates that it&amp;#039;s the default one&lt;br /&gt;
&lt;br /&gt;
:; AVAIL&lt;br /&gt;
:: state/availability of the partition: see [[#Partition availability|below]]&lt;br /&gt;
&lt;br /&gt;
:; STATE&lt;br /&gt;
:: state (using [https://slurm.schedmd.com/sinfo.html#SECTION_NODE-STATE-CODES these codes])&lt;br /&gt;
:: typical values are &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;mixed&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that some of the resources are busy executing jobs while other are idle, and &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;allocated&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that all of the resources are in use&lt;br /&gt;
&lt;br /&gt;
:; DEFAULTTIME&lt;br /&gt;
:: default runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:; TIMELIMIT&lt;br /&gt;
:: maximum runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
The asterisk at the end of the partition name indicates the default partition, i.e. the one on which jobs which do not ask for a specific partition are run.&lt;br /&gt;
&lt;br /&gt;
Command &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; does not tell you about the &amp;#039;&amp;#039;jobs&amp;#039;&amp;#039; submitted to a partition. This information is obtained, instead, with [[User Jobs#Inspecting jobs with squeue|command &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;]].&lt;br /&gt;
&lt;br /&gt;
== Partition availability ==&lt;br /&gt;
&lt;br /&gt;
The most important information that &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; provides is the &amp;#039;&amp;#039;&amp;#039;availability&amp;#039;&amp;#039;&amp;#039; (also called &amp;#039;&amp;#039;&amp;#039;state&amp;#039;&amp;#039;&amp;#039;) of partitions. This is shown in column &amp;quot;AVAIL&amp;quot;. Possible partition states are:&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is available&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed as soon as resources allow&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is in the process of becoming unavailable (i.e., of entering the &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; state: see below)&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is unavailable&lt;br /&gt;
:: There are no running jobs&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
When a partition goes from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; no harm is done to running jobs. In a normally functioning SLURM system, the passage from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; happens only when no jobs are running on the partition. If (e.g., due to a malfunction) the passage happens with jobs still running, they get killed.&lt;br /&gt;
&lt;br /&gt;
A partition in state &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; requires intervention by a [[Roles|Job Administrator]] to be restored to &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
== Default values ==&lt;br /&gt;
&lt;br /&gt;
The features of SLURM partitions, including the &amp;#039;&amp;#039;&amp;#039;default values&amp;#039;&amp;#039;&amp;#039; which are applied to jobs that do not make explicit requests, can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scontrol show partition&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PartitionName=jobs&lt;br /&gt;
   AllowGroups=ALL AllowAccounts=ALL AllowQos=nogpu,gpulight,gpu,gpuwide,gpuheavy-20,gpuheavy-40&lt;br /&gt;
   AllocNodes=ALL Default=YES QoS=N/A&lt;br /&gt;
   DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO&lt;br /&gt;
   MaxNodes=UNLIMITED MaxTime=3-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED&lt;br /&gt;
   Nodes=gn01&lt;br /&gt;
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO&lt;br /&gt;
   OverTimeLimit=NONE PreemptMode=OFF&lt;br /&gt;
   State=UP TotalCPUs=48 TotalNodes=1 SelectTypeParameters=NONE&lt;br /&gt;
   JobDefaults=(null)&lt;br /&gt;
   DefMemPerNode=4096 MaxMemPerNode=UNLIMITED&lt;br /&gt;
   TRES=cpu=48,mem=1011435M,node=1,billing=49,gres/gpu=13,gres/gpu:3g.20gb=5,gres/gpu:40gb=3,gres/gpu:4g.20gb=5&lt;br /&gt;
   TRESBillingWeights=cpu=1.0,gres/gpu:3g.20gb=6.0,gres/gpu:4g.20gb=6.0,gres/gpu:40gb=6.0,mem=0.05g&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In the output of &amp;lt;code&amp;gt;scontrol show partition&amp;lt;/code&amp;gt; the most relevant default values for Mufasa users are:&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefaultTime&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default execution time assigned to a job run on the partition (e.g., 1 hour)&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefMemPerNode&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default amount of RAM assigned to a job run on the partition (e.g., 4GB)&lt;br /&gt;
&lt;br /&gt;
= System resources subjected to limitations =&lt;br /&gt;
&lt;br /&gt;
In systems based on SLURM like Mufasa, &amp;#039;&amp;#039;&amp;#039;TRES (Trackable RESources)&amp;#039;&amp;#039;&amp;#039; are (from [https://slurm.schedmd.com/tres.html SLURM&amp;#039;s documentation] &amp;quot;&amp;#039;&amp;#039;resources that can be tracked for usage or used to enforce limits against.&amp;#039;&amp;#039;&amp;quot;&lt;br /&gt;
&lt;br /&gt;
TRES include CPUs, RAM and &amp;#039;&amp;#039;&amp;#039;GRES&amp;#039;&amp;#039;&amp;#039;. The last term stands for &amp;#039;&amp;#039;Generic RESources&amp;#039;&amp;#039; that a job may need for its execution. In Mufasa, the only &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; resources are the GPUs.&lt;br /&gt;
&lt;br /&gt;
== &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax ==&lt;br /&gt;
&lt;br /&gt;
To ask SLURM to assign GRES resources (i.e., GPUs) to a job, a special syntax must be used. Precisely, the name of each GPU resource takes the form&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:Name:Type&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
Considering the [[System#CPUs and GPUs|GPU complement of Mufasa]], &amp;lt;code&amp;gt;Type&amp;lt;/code&amp;gt; takes the following values:&lt;br /&gt;
&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:40gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 40 Gbytes of RAM&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:4g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 4 compute units&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 3 compute units&lt;br /&gt;
&lt;br /&gt;
So, for instance,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
identifies a resource corresponding to a GPU with 20 GB of RAM and 3 compute units.&lt;br /&gt;
&lt;br /&gt;
When asking for a GRES resource (e.g., in an &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command or an &amp;lt;code&amp;gt;SBATCH&amp;lt;/code&amp;gt; directive of an [[User Jobs#Using execution scripts to run jobs|execution script]]), the syntax required by SLURM is&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:&amp;lt;Type&amp;gt;:&amp;lt;Quantity&amp;gt;&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
where &amp;lt;code&amp;gt;Quantity&amp;lt;/code&amp;gt; is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of type &amp;lt;code&amp;gt;4g.20gb&amp;lt;/code&amp;gt; the syntax is&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:4g.20gb:2&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SLURM&amp;#039;s &amp;#039;&amp;#039;generic resources&amp;#039;&amp;#039; are defined in &amp;lt;code&amp;gt;/etc/slurm/gres.conf&amp;lt;/code&amp;gt;. In order to make GPUs available to SLURM&amp;#039;s &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; management, Mufasa makes use of Nvidia&amp;#039;s [https://developer.nvidia.com/nvidia-management-library-nvml NVML library]. For additional information see [https://slurm.schedmd.com/gres.html SLURM&amp;#039;s documentation].&lt;/div&gt;</summary>
		<author><name>GiulioFontana</name></author>
	</entry>
	<entry>
		<id>https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2605</id>
		<title>SLURM</title>
		<link rel="alternate" type="text/html" href="https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2605"/>
		<updated>2026-11-19T08:59:22Z</updated>

		<summary type="html">&lt;p&gt;GiulioFontana: /* SLURM in a nutshell */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page presents the features of SLURM that are most relevant to Mufasa&amp;#039;s [[Roles|Job Users]]. Job Users can submit jobs for execution, cancel their own jobs, and see other users&amp;#039; jobs (but not intervene on them).&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa &amp;#039;&amp;#039;&amp;#039;must use SLURM&amp;#039;&amp;#039;&amp;#039; to run resource-heavy processes, i.e. computing jobs that require one or more of the following:&lt;br /&gt;
* GPUs&lt;br /&gt;
* multiple CPUs&lt;br /&gt;
* powerful CPUs&lt;br /&gt;
* a significant amount of RAM&lt;br /&gt;
&lt;br /&gt;
In fact, only processes run via SLURM have access to all the resources of Mufasa. Processes run outside SLURM are executed by the [[System#Login server|login server]] virtual machine, which has minimal resources and no GPUs. Using SLURM is therefore the only way to execute resource-heavy jobs on Mufasa (this is a key difference between Mufasa 1.0 and Mufasa 2.0).&lt;br /&gt;
&lt;br /&gt;
= SLURM in a nutshell =&lt;br /&gt;
&lt;br /&gt;
Computation jobs on Mufasa needs to be launched via [[System#The SLURM job scheduling system|SLURM]]. SLURM provides jobs with access to the [[#System resources subjected to limitations|physical resources]] of Mufasa, such as CPUs, GPUs and RAM. Thanks to SLURM, processing jobs share system resources, optimising their occupation and availability. &lt;br /&gt;
&lt;br /&gt;
When a user runs a job, the job does not get executed immediately and is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039;. SLURM executes jobs according to their order in the queue: the top job in the queue gets executed as soon as the necessary resources are available, while jobs lower in the queue wait longer. The position of a job in the queue is due to the [[#Job priority|priority]] assigned to it by SLURM, with higher-priority jobs closer to the top. As a general rule:&lt;br /&gt;
&lt;br /&gt;
;: &amp;#039;&amp;#039;&amp;#039;the greater the fraction of Mufasa&amp;#039;s overall resources that a job asks for, the lower the job&amp;#039;s priority will be&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
The priority mechanism is used to encourage users to use Mufasa&amp;#039;s resources (i.e.: GPUs, CPUs, RAM, execution time) in an effective and equitable manner. This page includes a [[#How_to_maximise_the_priority_of_your_jobs|chart explaining how to maximise the priority of your jobs]].&lt;br /&gt;
&lt;br /&gt;
The &amp;#039;&amp;#039;&amp;#039;time&amp;#039;&amp;#039;&amp;#039; available to a job for its execution is controlled by SLURM. When a user requests execution of a job, they must specify the duration of the time slot that the job needs. The job must complete its execution before the end of the requested time slot, because at the end of such slot it gets killed by SLURM.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0 access to system resources is managed via SLURM&amp;#039;s [[#SLURM Quality of Service (QOS)|Quality of Service (QOS)]] mechanism (Mufasa 1.0 used [[#SLURM_partitions|partitions]] instead). While [[User Jobs#Running jobs with SLURM|launching a processing job via SLURM]], the user must always specify the chosen QOS. QOSes differ in the set of resources that they provide access to because each of them is designed to fit a given type of job.&lt;br /&gt;
&lt;br /&gt;
Mufasa sets limits to the number of jobs by the same user. This page includes a [[#Limits on jobs by the same user|table summarising such limits]].&lt;br /&gt;
&lt;br /&gt;
= SLURM Quality of Service (QOS) =&lt;br /&gt;
&lt;br /&gt;
Through &amp;#039;&amp;#039;&amp;#039;Quality of Services&amp;#039;&amp;#039;&amp;#039; (&amp;#039;&amp;#039;&amp;#039;QOSes&amp;#039;&amp;#039;&amp;#039;), SLURM lets system configurators assign a name to a set of related constraints.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, QOSes are used to define different levels of access to the server&amp;#039;s resources. When [[User Jobs|executing a job with SLURM]], a user must always &amp;#039;&amp;#039;&amp;#039;specify the QOS&amp;#039;&amp;#039;&amp;#039; that their job will use: this choice, in turn, determines what resources the job is able to access and influences the [[#Job priority|priority]] of the job.&lt;br /&gt;
&lt;br /&gt;
Mufasa&amp;#039;s QOSes and their features can be inspected with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list qos format=name%-11,priority,MaxSubmit,maxwall,maxtres%-80,mintres%-18&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following (see below for the same data in table form):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
Name          Priority MaxSubmit     MaxWall MaxTRES                                                                          MinTRES            &lt;br /&gt;
----------- ---------- --------- ----------- -------------------------------------------------------------------------------- ------------------ &lt;br /&gt;
normal               0                                                                                                                           &lt;br /&gt;
nogpu                4         1  3-00:00:00 cpu=16,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=128G                    &lt;br /&gt;
gpuheavy-20          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=2,mem=128G             gres/gpu:4g.20gb=1 &lt;br /&gt;
gpuheavy-40          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=1,gres/gpu:4g.20gb=0,mem=128G             gres/gpu:40gb=1    &lt;br /&gt;
gpulight             8         1    12:00:00 cpu=2,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpu                  2         1  1-00:00:00 cpu=8,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpuwide              2         2  1-00:00:00 cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=1,mem=64G              gres/gpu:4g.20gb=1 &lt;br /&gt;
build               32         1    02:00:00 cpu=2,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=16G&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The columns of this output are the following:&lt;br /&gt;
&lt;br /&gt;
:; Name&lt;br /&gt;
:: name of the QOS&lt;br /&gt;
&lt;br /&gt;
:; Priority&lt;br /&gt;
:: priority tier associated to the QOS (higher value = higher priority): see [[#Job priority|Job priority]] for details&lt;br /&gt;
&lt;br /&gt;
:; MaxSubmit&lt;br /&gt;
:: maximum number of jobs from a single user that can be submitted to SLURM with this QOS; submitted jobs include both running and queued jobs&lt;br /&gt;
:: See [[#Limits on jobs by the same user|Limits on jobs by the same user]] for an overview of the limits on jobs set by Mufasa.&lt;br /&gt;
&lt;br /&gt;
:; MaxWall&lt;br /&gt;
:: maximum wall clock duration of the jobs using the QOS (after which they are killed by SLURM), in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
:: For some QOSes these are not set: it means that they are determined by the [[#SLURM partitions|partition]]. Partitions also define the [[#Default values|default duration]] of jobs.&lt;br /&gt;
&lt;br /&gt;
:; MaxTRES&lt;br /&gt;
:: amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) available to a job using the QOS, where&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;cpu=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of CPUs (i.e., processor cores) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of CPUs specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;gres/&amp;#039;&amp;#039;gpu:Type&amp;#039;&amp;#039;=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of GPUs of class &amp;lt;code&amp;gt;&amp;#039;&amp;#039;Type&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; (see [[User Jobs#gres syntax|&amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax]]) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; (for QOSes that allow access to GPUs) if not specified, the job cannot be launched&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;mem=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;G&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum amount of system RAM is &amp;#039;&amp;#039;K&amp;#039;&amp;#039; GBytes&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of RAM specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
&lt;br /&gt;
:; MinTRES&lt;br /&gt;
:: minimum amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) that a job using the QOS must request in order to actually get executed by SLURM.&lt;br /&gt;
:: (If your job does not actually need these resources, you&amp;#039;ve chosen the wrong QOS: you can use one with a higher priority.)&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; QOS is the default one, and exists only to ensure that users always specify a QOS when running a job. Since &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; has zero priority and no resources, a job run using this QOS would never be run.&lt;br /&gt;
&lt;br /&gt;
The information provided by the &amp;lt;code&amp;gt;sacctmgr list qos&amp;lt;/code&amp;gt; command above is summarised by the following table:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! QOS&lt;br /&gt;
! Priority&amp;lt;/br&amp;gt;tier&lt;br /&gt;
! Max&amp;lt;/br&amp;gt;Submit&lt;br /&gt;
! MaxWall&amp;lt;/br&amp;gt;[h]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;CPUs&lt;br /&gt;
! max RAM&amp;lt;/br&amp;gt;[GB]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;3g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;4g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;40GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | build&lt;br /&gt;
| 32&lt;br /&gt;
| 1&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 16&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpulight&lt;br /&gt;
| 8&lt;br /&gt;
| 1&lt;br /&gt;
| 12&lt;br /&gt;
| 2&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | nogpu&lt;br /&gt;
| 4&lt;br /&gt;
| 1&lt;br /&gt;
| 72&lt;br /&gt;
| 16&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpu&lt;br /&gt;
| 2&lt;br /&gt;
| 1&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuwide&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-20&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| 2 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-40&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
A key piece of information in the table above is the &amp;#039;&amp;#039;&amp;#039;priority tier&amp;#039;&amp;#039;&amp;#039; associated to each QOS. In Mufasa 2.0, priority tiers are used to encourage users to use the &amp;#039;&amp;#039;&amp;#039;least powerful QOS that is compatible with their needs&amp;#039;&amp;#039;&amp;#039;, where &amp;quot;powerful&amp;quot; means &amp;quot;rich with resources&amp;quot;. Less powerful QOSes increase the priority of the jobs that use them, so these jobs tend to be executed sooner. See [[#Job priority|Job priority]] for details about how priority affects the execution order of jobs in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
== The &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS ==&lt;br /&gt;
&lt;br /&gt;
This QOS is specifically designed to be used by Mufasa users to &amp;#039;&amp;#039;&amp;#039;quickly build [[System#Containers|container images]]&amp;#039;&amp;#039;&amp;#039;. Its associated priority tier is very high: this, combined with the fact that jobs using this QOS require few resources, means that such jobs usually get executed very soon.&lt;br /&gt;
&lt;br /&gt;
On the other hand, the limited resources and the lack of access to the GPUs make the &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS unsuitable for other tasks.&lt;br /&gt;
&lt;br /&gt;
See [[Singularity#Building Singularity images|Building Singularity images]] for directions about building Singularity container images.&lt;br /&gt;
&lt;br /&gt;
== Restricted QOSes ==&lt;br /&gt;
&lt;br /&gt;
In Mufasa, the most powerful QOSes are reserved to researchers (called &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users: these include academic personnel and Ph.D. students). M.Sc. students, called &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users, cannot use them while running jobs.&lt;br /&gt;
&lt;br /&gt;
See [[#research users and students users|below]] to understand how &amp;#039;&amp;#039;user categories&amp;#039;&amp;#039; work in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
= &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users =&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa belong to two &amp;#039;&amp;#039;&amp;#039;user categories&amp;#039;&amp;#039;&amp;#039;. Categories provide users with different access to  to Mufasa&amp;#039;s resources: the idea being to provide researchers with more access while still letting students benefit from the server&amp;#039;s capabilities.&lt;br /&gt;
&lt;br /&gt;
User categories in Mufasa are the following:&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. academic personnel and Ph.D. students&lt;br /&gt;
::: * can use all QOSes, including the the [[#Restricted QOSes|restricted ones]]&lt;br /&gt;
::: * their jobs have a higher &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a higher number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. M.Sc. students&lt;br /&gt;
::: * cannot use the [[#Restricted QOSes|restricted QOSes]]&lt;br /&gt;
::: * their jobs have a lower &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a lower number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
You can inspect the differences in priority and running jobs between &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association format=&amp;quot;account,priority,maxjobs&amp;quot; | grep -E &amp;#039;Account|research|students&amp;#039;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
   Account   Priority MaxJobs &lt;br /&gt;
  research          4       2 &lt;br /&gt;
  students          1       1&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
To know what limits apply to your own user, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association where user=$USER format=&amp;quot;user,priority,maxjobs,qos%-60&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
      User   Priority MaxJobs QOS                                                          &lt;br /&gt;
---------- ---------- ------- ------------------------------------------------------------ &lt;br /&gt;
    preali          4       2 build,gpu,gpuheavy-20,gpuheavy-40,gpulight,gpuwide,nogpu&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The list under &amp;quot;QOS&amp;quot; shows what QOSes your user is allowed to use when [[User Jobs|running jobs]]. &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users can use all of them, while &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users can only access a subset of them.&lt;br /&gt;
&lt;br /&gt;
= Limits on jobs by the same user =&lt;br /&gt;
&lt;br /&gt;
Mufasa enforces limits on the number of jobs from a single user. Such limits aim at preventing users from &amp;quot;hogging&amp;quot; system resources, and apply to:&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;submitted jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that the user asked SLURM to execute, each of which may currently be either running or queued&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;running jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that are currently in execution&lt;br /&gt;
&lt;br /&gt;
Limits on submitted jobs are set via [[#SLURM Quality of Service (QOS)|QOSes]], while limits on running jobs are set via [[#research users and students users|user category]].&lt;br /&gt;
&lt;br /&gt;
The following table summarises what limits exist:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! &lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;running&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;submitted&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | global limits&amp;lt;br/&amp;gt;(system-wide)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to limits set by individual QOSes (below)&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | limits for&amp;lt;br/&amp;gt;individual&amp;lt;/br&amp;gt;QOSes&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to global limits (above)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for the&amp;#039;&amp;#039; &amp;lt;code&amp;gt;gpuwide&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;QOS&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for each of the other QOSes&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
= Job priority =&lt;br /&gt;
&lt;br /&gt;
Once the execution of a job has been requested, the job is not run immediately: it is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039; by SLURM, together with all the other jobs awaiting execution. The job on top of the queue at any time is the first to be put into execution as soon as the resources it requires are available. &lt;br /&gt;
&lt;br /&gt;
The order of the items in the job queue depends on their &amp;#039;&amp;#039;&amp;#039;priority&amp;#039;&amp;#039;&amp;#039;. The job with the highest priority is put by SLURM on top of the queue, while all other jobs are put in the queue in order of descending priority.&lt;br /&gt;
&lt;br /&gt;
The goal of SLURM is to maximise resource availability: i.e., to ensure the shorter possible wait time before job execution. To achieve this goal, in Mufasa SLURM is configured to &amp;#039;&amp;#039;&amp;#039;encourage users not to ask for resources or execution time that their job doesn&amp;#039;t need&amp;#039;&amp;#039;&amp;#039;. This is done via the priority mechanism: the more resources and/or time a job requests, the lower its priority will be; and the later it will be executed.&lt;br /&gt;
&lt;br /&gt;
Priority management in Mufasa is designed to set up a &amp;#039;&amp;#039;&amp;#039;virtuous cycle&amp;#039;&amp;#039;&amp;#039; where users, by carefully choosing what to ask for, obtain two results:&lt;br /&gt;
* they ensure that their job is executed as soon as possible;&lt;br /&gt;
* they leave as much as possible of Mufasa&amp;#039;s resources free for other users&amp;#039;s jobs.&lt;br /&gt;
&lt;br /&gt;
[[#How to maximise the priority of your jobs|Useful tips]] to maximise the priority of your jobs are provided below.&lt;br /&gt;
&lt;br /&gt;
== Elements determining job priority ==&lt;br /&gt;
In Mufasa, the priority of a job is computed by SLURM according to the following elements:&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#research users and students users|User category]]&amp;#039;&amp;#039;&amp;#039; (i.e., &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;)&lt;br /&gt;
::: Used to provide higher priority to jobs run by &amp;#039;&amp;#039;&amp;#039;research personnel&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#SLURM Quality of Service (QOS)|QOS]]&amp;#039;&amp;#039;&amp;#039; used by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;less resources&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Number of CPUs&amp;#039;&amp;#039;&amp;#039; requested by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;a lower number of CPUs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job Age&amp;#039;&amp;#039;&amp;#039;, i.e. the time that the job has been waiting in the queue&lt;br /&gt;
::: Used to provide higher priority to jobs which have been &amp;#039;&amp;#039;&amp;#039;queued for a longer time&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
::: (Periods when a queued job is not eligible for running because of the [[#Limits on jobs by the same user|limits on running jobs by the same user]] do not increase job age)&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;FairShare&amp;#039;&amp;#039;&amp;#039;, i.e. a factor computed by SLURM to balance use of the system by different users&lt;br /&gt;
::: Used to provide higher priority to jobs by users who &amp;#039;&amp;#039;&amp;#039;used Mufasa&amp;#039;s resources less than others&amp;#039;&amp;#039;&amp;#039; (see [[#How FairShare works|below]])&lt;br /&gt;
&lt;br /&gt;
You can understand how each of these elements contributes to the priority of the jobs waiting for execution (both yours and of other users) with command &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;. A tailored version of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; that only displays the parameters listed above is the following:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sprio -o &amp;quot;%6i %9u %.8Y %.10B %.5Q %.9T %.5A %.5F&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Here is an example of the output of the command above:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
JOBID  USER      PRIORITY      ASSOC   QOS      TRES   AGE FAIRS&lt;br /&gt;
5790   amoglia       3495       2500   375  cpu=-800   933   487&lt;br /&gt;
5791   qliu          3010       2500   375  cpu=-800   916    19&lt;br /&gt;
5817   mzoccola      2336        625   750  cpu=-800   495  1266&lt;br /&gt;
5822   fbottazzi     2235        625   750  cpu=-600   467   994&lt;br /&gt;
5837   lpassoni      2508       2500   750  cpu=-800     0    58&lt;br /&gt;
5838   bagostini     4437        625  3000  cpu=-200     0  1013&lt;br /&gt;
5839   flazzaro      1549        625   750  cpu=-800     0   974&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
In it, column PRIORITY provides the overall priority of each job. Each priority value is the sum of the five values to its right; in particular, column ASSOC corresponds to user category, while column TRES corresponds to the number of CPUs. Note that the contribution of TRES to priority is &amp;#039;&amp;#039;negative&amp;#039;&amp;#039;, i.e. it is a penalty; its value is proportional to the number of CPUs requested by the job.&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;Things to consider when using &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
* Its output is only relevant for jobs &amp;#039;&amp;#039;not yet in execution&amp;#039;&amp;#039;. For an already running job, the values shown by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; correspond to the priority that the job would get if evaluated now: however, this priority does not affect job execution in any way.&lt;br /&gt;
* Jobs listed by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; may temporarily differ from jobs listed by &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;: this is due to the fact that SLURM updates its internal data (used by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;) every few minutes, while the output of &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt; is updated much more frequently.&lt;br /&gt;
&lt;br /&gt;
=== How FairShare works ===&lt;br /&gt;
In Mufasa 2.0, the FairShare of each user is computed according to these rules:&lt;br /&gt;
# It considers the quantity used by (current and past) user jobs of:&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;CPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of CPUs used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;RAM&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the amount of RAM used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;GPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of GPUs used&lt;br /&gt;
#** note: 40 GB GPUs have double the impact on FairShare than 20 GB GPUs&lt;br /&gt;
# The impact on FairShare of any use of resources is &amp;#039;&amp;#039;&amp;#039;proportional to its duration&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
#* example: using 32 GB of RAM for 48h has the same impact of using 64 GB of RAM for 24h&lt;br /&gt;
# FairShare has a &amp;quot;fading memory&amp;quot;: i.e., resource use has less and less impact on FairShare the farther in the past it is&lt;br /&gt;
#* consequence: over time, the &amp;quot;history&amp;quot; of a user gets forgotten by FairShare&lt;br /&gt;
&lt;br /&gt;
== How to maximise the priority of your jobs ==&lt;br /&gt;
&lt;br /&gt;
Every time you run a SLURM job, follow these guidelines:&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Choose the less powerful QOS compatible with the needs of your job&lt;br /&gt;
:: QOSes with access to less resources have higher base priority.&lt;br /&gt;
&lt;br /&gt;
; Only request CPUs and RAM that your job really needs&lt;br /&gt;
:: Usually, code exploits multiple CPUs only if specifically designed to do so: if unsure, only ask for 1 CPU.&lt;br /&gt;
:: The fewer resources your job asks for, the less it will wait before they become available.&lt;br /&gt;
:: The more CPUs your job asks for, the stronger the penalty to its priority.&lt;br /&gt;
:: Asking for fewer CPUs and/or less RAM improves your FairShare.&lt;br /&gt;
&lt;br /&gt;
; Do not request more time than your jobs needs to complete&lt;br /&gt;
:: Make a worst-case estimate and only ask for that duration.&lt;br /&gt;
:: Asking for less of Mufasa&amp;#039;s time improves your FairShare.&lt;br /&gt;
&lt;br /&gt;
; Debug and test code using less powerful QOSes before running it with more powerful QOSes&lt;br /&gt;
:: Your test jobs will get a higher priority and your FairShare will improve.&lt;br /&gt;
&lt;br /&gt;
; Cancel jobs when you don&amp;#039;t need them anymore&lt;br /&gt;
:: Always use [[User_Jobs#Cancelling_a_job_with_scancel|scancel]] to delete completed (or crashed) jobs: your Fairshare will improve.&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Protip: it&amp;#039;s a good idea to [[User Jobs#Looking for unused GPUs|check for unused GPUs]] before choosing what to request. Requesting a GPU that is currently idle will help your job get executed sooner.&lt;br /&gt;
&lt;br /&gt;
= SLURM partitions =&lt;br /&gt;
&lt;br /&gt;
Partitions are another mechanism provided by SLURM to create different levels of access to system resources. Since in Mufasa 2.0 access to resources is controlled via [[#SLURM Quality of Service (QOS)|QOSes]], partitions are not very relevant. (This is one of the main differences between Mufasa 1.0 and Mufasa 2.0.)&lt;br /&gt;
&lt;br /&gt;
Note, however, that the default values for some features of SLURM jobs (e.g., duration) are [[#Default values|set by the partition]].&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, there is a single SLURM partition, called &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt;, and all jobs run on it. The state of &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt; can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sinfo -o &amp;quot;%10P %5a %9T %11L %10l&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PARTITION  AVAIL STATE     DEFAULTTIME TIMELIMIT &lt;br /&gt;
jobs*      up    idle      1:00:00     3-00:00:00&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where columns correspond to the following information:&lt;br /&gt;
&lt;br /&gt;
:; PARTITION&lt;br /&gt;
:: name of the partition; the asterisks indicates that it&amp;#039;s the default one&lt;br /&gt;
&lt;br /&gt;
:; AVAIL&lt;br /&gt;
:: state/availability of the partition: see [[#Partition availability|below]]&lt;br /&gt;
&lt;br /&gt;
:; STATE&lt;br /&gt;
:: state (using [https://slurm.schedmd.com/sinfo.html#SECTION_NODE-STATE-CODES these codes])&lt;br /&gt;
:: typical values are &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;mixed&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that some of the resources are busy executing jobs while other are idle, and &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;allocated&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that all of the resources are in use&lt;br /&gt;
&lt;br /&gt;
:; DEFAULTTIME&lt;br /&gt;
:: default runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:; TIMELIMIT&lt;br /&gt;
:: maximum runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
The asterisk at the end of the partition name indicates the default partition, i.e. the one on which jobs which do not ask for a specific partition are run.&lt;br /&gt;
&lt;br /&gt;
Command &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; does not tell you about the &amp;#039;&amp;#039;jobs&amp;#039;&amp;#039; submitted to a partition. This information is obtained, instead, with [[User Jobs#Inspecting jobs with squeue|command &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;]].&lt;br /&gt;
&lt;br /&gt;
== Partition availability ==&lt;br /&gt;
&lt;br /&gt;
The most important information that &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; provides is the &amp;#039;&amp;#039;&amp;#039;availability&amp;#039;&amp;#039;&amp;#039; (also called &amp;#039;&amp;#039;&amp;#039;state&amp;#039;&amp;#039;&amp;#039;) of partitions. This is shown in column &amp;quot;AVAIL&amp;quot;. Possible partition states are:&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is available&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed as soon as resources allow&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is in the process of becoming unavailable (i.e., of entering the &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; state: see below)&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is unavailable&lt;br /&gt;
:: There are no running jobs&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
When a partition goes from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; no harm is done to running jobs. In a normally functioning SLURM system, the passage from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; happens only when no jobs are running on the partition. If (e.g., due to a malfunction) the passage happens with jobs still running, they get killed.&lt;br /&gt;
&lt;br /&gt;
A partition in state &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; requires intervention by a [[Roles|Job Administrator]] to be restored to &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
== Default values ==&lt;br /&gt;
&lt;br /&gt;
The features of SLURM partitions, including the &amp;#039;&amp;#039;&amp;#039;default values&amp;#039;&amp;#039;&amp;#039; which are applied to jobs that do not make explicit requests, can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scontrol show partition&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PartitionName=jobs&lt;br /&gt;
   AllowGroups=ALL AllowAccounts=ALL AllowQos=nogpu,gpulight,gpu,gpuwide,gpuheavy-20,gpuheavy-40&lt;br /&gt;
   AllocNodes=ALL Default=YES QoS=N/A&lt;br /&gt;
   DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO&lt;br /&gt;
   MaxNodes=UNLIMITED MaxTime=3-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED&lt;br /&gt;
   Nodes=gn01&lt;br /&gt;
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO&lt;br /&gt;
   OverTimeLimit=NONE PreemptMode=OFF&lt;br /&gt;
   State=UP TotalCPUs=48 TotalNodes=1 SelectTypeParameters=NONE&lt;br /&gt;
   JobDefaults=(null)&lt;br /&gt;
   DefMemPerNode=4096 MaxMemPerNode=UNLIMITED&lt;br /&gt;
   TRES=cpu=48,mem=1011435M,node=1,billing=49,gres/gpu=13,gres/gpu:3g.20gb=5,gres/gpu:40gb=3,gres/gpu:4g.20gb=5&lt;br /&gt;
   TRESBillingWeights=cpu=1.0,gres/gpu:3g.20gb=6.0,gres/gpu:4g.20gb=6.0,gres/gpu:40gb=6.0,mem=0.05g&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In the output of &amp;lt;code&amp;gt;scontrol show partition&amp;lt;/code&amp;gt; the most relevant default values for Mufasa users are:&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefaultTime&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default execution time assigned to a job run on the partition (e.g., 1 hour)&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefMemPerNode&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default amount of RAM assigned to a job run on the partition (e.g., 4GB)&lt;br /&gt;
&lt;br /&gt;
= System resources subjected to limitations =&lt;br /&gt;
&lt;br /&gt;
In systems based on SLURM like Mufasa, &amp;#039;&amp;#039;&amp;#039;TRES (Trackable RESources)&amp;#039;&amp;#039;&amp;#039; are (from [https://slurm.schedmd.com/tres.html SLURM&amp;#039;s documentation] &amp;quot;&amp;#039;&amp;#039;resources that can be tracked for usage or used to enforce limits against.&amp;#039;&amp;#039;&amp;quot;&lt;br /&gt;
&lt;br /&gt;
TRES include CPUs, RAM and &amp;#039;&amp;#039;&amp;#039;GRES&amp;#039;&amp;#039;&amp;#039;. The last term stands for &amp;#039;&amp;#039;Generic RESources&amp;#039;&amp;#039; that a job may need for its execution. In Mufasa, the only &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; resources are the GPUs.&lt;br /&gt;
&lt;br /&gt;
== &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax ==&lt;br /&gt;
&lt;br /&gt;
To ask SLURM to assign GRES resources (i.e., GPUs) to a job, a special syntax must be used. Precisely, the name of each GPU resource takes the form&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:Name:Type&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
Considering the [[System#CPUs and GPUs|GPU complement of Mufasa]], &amp;lt;code&amp;gt;Type&amp;lt;/code&amp;gt; takes the following values:&lt;br /&gt;
&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:40gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 40 Gbytes of RAM&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:4g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 4 compute units&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 3 compute units&lt;br /&gt;
&lt;br /&gt;
So, for instance,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
identifies a resource corresponding to a GPU with 20 GB of RAM and 3 compute units.&lt;br /&gt;
&lt;br /&gt;
When asking for a GRES resource (e.g., in an &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command or an &amp;lt;code&amp;gt;SBATCH&amp;lt;/code&amp;gt; directive of an [[User Jobs#Using execution scripts to run jobs|execution script]]), the syntax required by SLURM is&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:&amp;lt;Type&amp;gt;:&amp;lt;Quantity&amp;gt;&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
where &amp;lt;code&amp;gt;Quantity&amp;lt;/code&amp;gt; is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of type &amp;lt;code&amp;gt;4g.20gb&amp;lt;/code&amp;gt; the syntax is&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:4g.20gb:2&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SLURM&amp;#039;s &amp;#039;&amp;#039;generic resources&amp;#039;&amp;#039; are defined in &amp;lt;code&amp;gt;/etc/slurm/gres.conf&amp;lt;/code&amp;gt;. In order to make GPUs available to SLURM&amp;#039;s &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; management, Mufasa makes use of Nvidia&amp;#039;s [https://developer.nvidia.com/nvidia-management-library-nvml NVML library]. For additional information see [https://slurm.schedmd.com/gres.html SLURM&amp;#039;s documentation].&lt;/div&gt;</summary>
		<author><name>GiulioFontana</name></author>
	</entry>
	<entry>
		<id>https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2604</id>
		<title>SLURM</title>
		<link rel="alternate" type="text/html" href="https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2604"/>
		<updated>2026-11-19T08:57:48Z</updated>

		<summary type="html">&lt;p&gt;GiulioFontana: /* Job priority */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page presents the features of SLURM that are most relevant to Mufasa&amp;#039;s [[Roles|Job Users]]. Job Users can submit jobs for execution, cancel their own jobs, and see other users&amp;#039; jobs (but not intervene on them).&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa &amp;#039;&amp;#039;&amp;#039;must use SLURM&amp;#039;&amp;#039;&amp;#039; to run resource-heavy processes, i.e. computing jobs that require one or more of the following:&lt;br /&gt;
* GPUs&lt;br /&gt;
* multiple CPUs&lt;br /&gt;
* powerful CPUs&lt;br /&gt;
* a significant amount of RAM&lt;br /&gt;
&lt;br /&gt;
In fact, only processes run via SLURM have access to all the resources of Mufasa. Processes run outside SLURM are executed by the [[System#Login server|login server]] virtual machine, which has minimal resources and no GPUs. Using SLURM is therefore the only way to execute resource-heavy jobs on Mufasa (this is a key difference between Mufasa 1.0 and Mufasa 2.0).&lt;br /&gt;
&lt;br /&gt;
= SLURM in a nutshell =&lt;br /&gt;
&lt;br /&gt;
Computation jobs on Mufasa needs to be launched via [[System#The SLURM job scheduling system|SLURM]]. SLURM provides jobs with access to the [[#System resources subjected to limitations|physical resources]] of Mufasa, such as CPUs, GPUs and RAM. Thanks to SLURM, processing jobs share system resources, optimising their occupation and availability. &lt;br /&gt;
&lt;br /&gt;
When a user runs a job, the job does not get executed immediately and is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039;. SLURM executes jobs according to their order in the queue: the top job in the queue gets executed as soon as the necessary resources are available, while jobs lower in the queue wait longer. The position of a job in the queue is due to the &amp;#039;&amp;#039;&amp;#039;[[#Job priority|priority]]&amp;#039;&amp;#039;&amp;#039; assigned to it by SLURM, with higher-priority jobs closer to the top. As a general rule:&lt;br /&gt;
&lt;br /&gt;
;: &amp;#039;&amp;#039;&amp;#039;the greater the fraction of Mufasa&amp;#039;s overall resources that a job asks for, the lower the job&amp;#039;s priority will be&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
The priority mechanism is used to encourage users to use Mufasa&amp;#039;s resources (i.e.: GPUs, CPUs, RAM, execution time) in an effective and equitable manner. This page includes a [[#How_to_maximise_the_priority_of_your_jobs|chart explaining how to maximise the priority of your jobs]].&lt;br /&gt;
&lt;br /&gt;
The &amp;#039;&amp;#039;&amp;#039;time&amp;#039;&amp;#039;&amp;#039; available to a job for its execution is controlled by SLURM. When a user requests execution of a job, they must specify the duration of the time slot that the job needs. The job must complete its execution before the end of the requested time slot, because at the end of such slot it gets killed by SLURM.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0 access to system resources is managed via SLURM&amp;#039;s [[#SLURM Quality of Service (QOS)|Quality of Service (QOS)]] mechanism (Mufasa 1.0 used [[#SLURM_partitions|partitions]] instead). While [[User Jobs#Running jobs with SLURM|launching a processing job via SLURM]], the user must always specify the chosen QOS. QOSes differ in the set of resources that they provide access to because each of them is designed to fit a given type of job.&lt;br /&gt;
&lt;br /&gt;
Mufasa sets limits to the number of jobs by the same user. This page includes a [[#Limits on jobs by the same user|table summarising such limits]].&lt;br /&gt;
&lt;br /&gt;
= SLURM Quality of Service (QOS) =&lt;br /&gt;
&lt;br /&gt;
Through &amp;#039;&amp;#039;&amp;#039;Quality of Services&amp;#039;&amp;#039;&amp;#039; (&amp;#039;&amp;#039;&amp;#039;QOSes&amp;#039;&amp;#039;&amp;#039;), SLURM lets system configurators assign a name to a set of related constraints.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, QOSes are used to define different levels of access to the server&amp;#039;s resources. When [[User Jobs|executing a job with SLURM]], a user must always &amp;#039;&amp;#039;&amp;#039;specify the QOS&amp;#039;&amp;#039;&amp;#039; that their job will use: this choice, in turn, determines what resources the job is able to access and influences the [[#Job priority|priority]] of the job.&lt;br /&gt;
&lt;br /&gt;
Mufasa&amp;#039;s QOSes and their features can be inspected with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list qos format=name%-11,priority,MaxSubmit,maxwall,maxtres%-80,mintres%-18&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following (see below for the same data in table form):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
Name          Priority MaxSubmit     MaxWall MaxTRES                                                                          MinTRES            &lt;br /&gt;
----------- ---------- --------- ----------- -------------------------------------------------------------------------------- ------------------ &lt;br /&gt;
normal               0                                                                                                                           &lt;br /&gt;
nogpu                4         1  3-00:00:00 cpu=16,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=128G                    &lt;br /&gt;
gpuheavy-20          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=2,mem=128G             gres/gpu:4g.20gb=1 &lt;br /&gt;
gpuheavy-40          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=1,gres/gpu:4g.20gb=0,mem=128G             gres/gpu:40gb=1    &lt;br /&gt;
gpulight             8         1    12:00:00 cpu=2,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpu                  2         1  1-00:00:00 cpu=8,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpuwide              2         2  1-00:00:00 cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=1,mem=64G              gres/gpu:4g.20gb=1 &lt;br /&gt;
build               32         1    02:00:00 cpu=2,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=16G&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The columns of this output are the following:&lt;br /&gt;
&lt;br /&gt;
:; Name&lt;br /&gt;
:: name of the QOS&lt;br /&gt;
&lt;br /&gt;
:; Priority&lt;br /&gt;
:: priority tier associated to the QOS (higher value = higher priority): see [[#Job priority|Job priority]] for details&lt;br /&gt;
&lt;br /&gt;
:; MaxSubmit&lt;br /&gt;
:: maximum number of jobs from a single user that can be submitted to SLURM with this QOS; submitted jobs include both running and queued jobs&lt;br /&gt;
:: See [[#Limits on jobs by the same user|Limits on jobs by the same user]] for an overview of the limits on jobs set by Mufasa.&lt;br /&gt;
&lt;br /&gt;
:; MaxWall&lt;br /&gt;
:: maximum wall clock duration of the jobs using the QOS (after which they are killed by SLURM), in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
:: For some QOSes these are not set: it means that they are determined by the [[#SLURM partitions|partition]]. Partitions also define the [[#Default values|default duration]] of jobs.&lt;br /&gt;
&lt;br /&gt;
:; MaxTRES&lt;br /&gt;
:: amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) available to a job using the QOS, where&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;cpu=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of CPUs (i.e., processor cores) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of CPUs specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;gres/&amp;#039;&amp;#039;gpu:Type&amp;#039;&amp;#039;=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of GPUs of class &amp;lt;code&amp;gt;&amp;#039;&amp;#039;Type&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; (see [[User Jobs#gres syntax|&amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax]]) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; (for QOSes that allow access to GPUs) if not specified, the job cannot be launched&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;mem=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;G&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum amount of system RAM is &amp;#039;&amp;#039;K&amp;#039;&amp;#039; GBytes&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of RAM specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
&lt;br /&gt;
:; MinTRES&lt;br /&gt;
:: minimum amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) that a job using the QOS must request in order to actually get executed by SLURM.&lt;br /&gt;
:: (If your job does not actually need these resources, you&amp;#039;ve chosen the wrong QOS: you can use one with a higher priority.)&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; QOS is the default one, and exists only to ensure that users always specify a QOS when running a job. Since &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; has zero priority and no resources, a job run using this QOS would never be run.&lt;br /&gt;
&lt;br /&gt;
The information provided by the &amp;lt;code&amp;gt;sacctmgr list qos&amp;lt;/code&amp;gt; command above is summarised by the following table:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! QOS&lt;br /&gt;
! Priority&amp;lt;/br&amp;gt;tier&lt;br /&gt;
! Max&amp;lt;/br&amp;gt;Submit&lt;br /&gt;
! MaxWall&amp;lt;/br&amp;gt;[h]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;CPUs&lt;br /&gt;
! max RAM&amp;lt;/br&amp;gt;[GB]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;3g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;4g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;40GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | build&lt;br /&gt;
| 32&lt;br /&gt;
| 1&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 16&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpulight&lt;br /&gt;
| 8&lt;br /&gt;
| 1&lt;br /&gt;
| 12&lt;br /&gt;
| 2&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | nogpu&lt;br /&gt;
| 4&lt;br /&gt;
| 1&lt;br /&gt;
| 72&lt;br /&gt;
| 16&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpu&lt;br /&gt;
| 2&lt;br /&gt;
| 1&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuwide&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-20&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| 2 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-40&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
A key piece of information in the table above is the &amp;#039;&amp;#039;&amp;#039;priority tier&amp;#039;&amp;#039;&amp;#039; associated to each QOS. In Mufasa 2.0, priority tiers are used to encourage users to use the &amp;#039;&amp;#039;&amp;#039;least powerful QOS that is compatible with their needs&amp;#039;&amp;#039;&amp;#039;, where &amp;quot;powerful&amp;quot; means &amp;quot;rich with resources&amp;quot;. Less powerful QOSes increase the priority of the jobs that use them, so these jobs tend to be executed sooner. See [[#Job priority|Job priority]] for details about how priority affects the execution order of jobs in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
== The &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS ==&lt;br /&gt;
&lt;br /&gt;
This QOS is specifically designed to be used by Mufasa users to &amp;#039;&amp;#039;&amp;#039;quickly build [[System#Containers|container images]]&amp;#039;&amp;#039;&amp;#039;. Its associated priority tier is very high: this, combined with the fact that jobs using this QOS require few resources, means that such jobs usually get executed very soon.&lt;br /&gt;
&lt;br /&gt;
On the other hand, the limited resources and the lack of access to the GPUs make the &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS unsuitable for other tasks.&lt;br /&gt;
&lt;br /&gt;
See [[Singularity#Building Singularity images|Building Singularity images]] for directions about building Singularity container images.&lt;br /&gt;
&lt;br /&gt;
== Restricted QOSes ==&lt;br /&gt;
&lt;br /&gt;
In Mufasa, the most powerful QOSes are reserved to researchers (called &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users: these include academic personnel and Ph.D. students). M.Sc. students, called &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users, cannot use them while running jobs.&lt;br /&gt;
&lt;br /&gt;
See [[#research users and students users|below]] to understand how &amp;#039;&amp;#039;user categories&amp;#039;&amp;#039; work in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
= &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users =&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa belong to two &amp;#039;&amp;#039;&amp;#039;user categories&amp;#039;&amp;#039;&amp;#039;. Categories provide users with different access to  to Mufasa&amp;#039;s resources: the idea being to provide researchers with more access while still letting students benefit from the server&amp;#039;s capabilities.&lt;br /&gt;
&lt;br /&gt;
User categories in Mufasa are the following:&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. academic personnel and Ph.D. students&lt;br /&gt;
::: * can use all QOSes, including the the [[#Restricted QOSes|restricted ones]]&lt;br /&gt;
::: * their jobs have a higher &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a higher number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. M.Sc. students&lt;br /&gt;
::: * cannot use the [[#Restricted QOSes|restricted QOSes]]&lt;br /&gt;
::: * their jobs have a lower &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a lower number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
You can inspect the differences in priority and running jobs between &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association format=&amp;quot;account,priority,maxjobs&amp;quot; | grep -E &amp;#039;Account|research|students&amp;#039;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
   Account   Priority MaxJobs &lt;br /&gt;
  research          4       2 &lt;br /&gt;
  students          1       1&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
To know what limits apply to your own user, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association where user=$USER format=&amp;quot;user,priority,maxjobs,qos%-60&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
      User   Priority MaxJobs QOS                                                          &lt;br /&gt;
---------- ---------- ------- ------------------------------------------------------------ &lt;br /&gt;
    preali          4       2 build,gpu,gpuheavy-20,gpuheavy-40,gpulight,gpuwide,nogpu&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The list under &amp;quot;QOS&amp;quot; shows what QOSes your user is allowed to use when [[User Jobs|running jobs]]. &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users can use all of them, while &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users can only access a subset of them.&lt;br /&gt;
&lt;br /&gt;
= Limits on jobs by the same user =&lt;br /&gt;
&lt;br /&gt;
Mufasa enforces limits on the number of jobs from a single user. Such limits aim at preventing users from &amp;quot;hogging&amp;quot; system resources, and apply to:&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;submitted jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that the user asked SLURM to execute, each of which may currently be either running or queued&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;running jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that are currently in execution&lt;br /&gt;
&lt;br /&gt;
Limits on submitted jobs are set via [[#SLURM Quality of Service (QOS)|QOSes]], while limits on running jobs are set via [[#research users and students users|user category]].&lt;br /&gt;
&lt;br /&gt;
The following table summarises what limits exist:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! &lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;running&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;submitted&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | global limits&amp;lt;br/&amp;gt;(system-wide)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to limits set by individual QOSes (below)&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | limits for&amp;lt;br/&amp;gt;individual&amp;lt;/br&amp;gt;QOSes&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to global limits (above)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for the&amp;#039;&amp;#039; &amp;lt;code&amp;gt;gpuwide&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;QOS&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for each of the other QOSes&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
= Job priority =&lt;br /&gt;
&lt;br /&gt;
Once the execution of a job has been requested, the job is not run immediately: it is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039; by SLURM, together with all the other jobs awaiting execution. The job on top of the queue at any time is the first to be put into execution as soon as the resources it requires are available. &lt;br /&gt;
&lt;br /&gt;
The order of the items in the job queue depends on their &amp;#039;&amp;#039;&amp;#039;priority&amp;#039;&amp;#039;&amp;#039;. The job with the highest priority is put by SLURM on top of the queue, while all other jobs are put in the queue in order of descending priority.&lt;br /&gt;
&lt;br /&gt;
The goal of SLURM is to maximise resource availability: i.e., to ensure the shorter possible wait time before job execution. To achieve this goal, in Mufasa SLURM is configured to &amp;#039;&amp;#039;&amp;#039;encourage users not to ask for resources or execution time that their job doesn&amp;#039;t need&amp;#039;&amp;#039;&amp;#039;. This is done via the priority mechanism: the more resources and/or time a job requests, the lower its priority will be; and the later it will be executed.&lt;br /&gt;
&lt;br /&gt;
Priority management in Mufasa is designed to set up a &amp;#039;&amp;#039;&amp;#039;virtuous cycle&amp;#039;&amp;#039;&amp;#039; where users, by carefully choosing what to ask for, obtain two results:&lt;br /&gt;
* they ensure that their job is executed as soon as possible;&lt;br /&gt;
* they leave as much as possible of Mufasa&amp;#039;s resources free for other users&amp;#039;s jobs.&lt;br /&gt;
&lt;br /&gt;
[[#How to maximise the priority of your jobs|Useful tips]] to maximise the priority of your jobs are provided below.&lt;br /&gt;
&lt;br /&gt;
== Elements determining job priority ==&lt;br /&gt;
In Mufasa, the priority of a job is computed by SLURM according to the following elements:&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#research users and students users|User category]]&amp;#039;&amp;#039;&amp;#039; (i.e., &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;)&lt;br /&gt;
::: Used to provide higher priority to jobs run by &amp;#039;&amp;#039;&amp;#039;research personnel&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#SLURM Quality of Service (QOS)|QOS]]&amp;#039;&amp;#039;&amp;#039; used by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;less resources&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Number of CPUs&amp;#039;&amp;#039;&amp;#039; requested by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;a lower number of CPUs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job Age&amp;#039;&amp;#039;&amp;#039;, i.e. the time that the job has been waiting in the queue&lt;br /&gt;
::: Used to provide higher priority to jobs which have been &amp;#039;&amp;#039;&amp;#039;queued for a longer time&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
::: (Periods when a queued job is not eligible for running because of the [[#Limits on jobs by the same user|limits on running jobs by the same user]] do not increase job age)&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;FairShare&amp;#039;&amp;#039;&amp;#039;, i.e. a factor computed by SLURM to balance use of the system by different users&lt;br /&gt;
::: Used to provide higher priority to jobs by users who &amp;#039;&amp;#039;&amp;#039;used Mufasa&amp;#039;s resources less than others&amp;#039;&amp;#039;&amp;#039; (see [[#How FairShare works|below]])&lt;br /&gt;
&lt;br /&gt;
You can understand how each of these elements contributes to the priority of the jobs waiting for execution (both yours and of other users) with command &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;. A tailored version of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; that only displays the parameters listed above is the following:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sprio -o &amp;quot;%6i %9u %.8Y %.10B %.5Q %.9T %.5A %.5F&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Here is an example of the output of the command above:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
JOBID  USER      PRIORITY      ASSOC   QOS      TRES   AGE FAIRS&lt;br /&gt;
5790   amoglia       3495       2500   375  cpu=-800   933   487&lt;br /&gt;
5791   qliu          3010       2500   375  cpu=-800   916    19&lt;br /&gt;
5817   mzoccola      2336        625   750  cpu=-800   495  1266&lt;br /&gt;
5822   fbottazzi     2235        625   750  cpu=-600   467   994&lt;br /&gt;
5837   lpassoni      2508       2500   750  cpu=-800     0    58&lt;br /&gt;
5838   bagostini     4437        625  3000  cpu=-200     0  1013&lt;br /&gt;
5839   flazzaro      1549        625   750  cpu=-800     0   974&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
In it, column PRIORITY provides the overall priority of each job. Each priority value is the sum of the five values to its right; in particular, column ASSOC corresponds to user category, while column TRES corresponds to the number of CPUs. Note that the contribution of TRES to priority is &amp;#039;&amp;#039;negative&amp;#039;&amp;#039;, i.e. it is a penalty; its value is proportional to the number of CPUs requested by the job.&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;Things to consider when using &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
* Its output is only relevant for jobs &amp;#039;&amp;#039;not yet in execution&amp;#039;&amp;#039;. For an already running job, the values shown by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; correspond to the priority that the job would get if evaluated now: however, this priority does not affect job execution in any way.&lt;br /&gt;
* Jobs listed by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; may temporarily differ from jobs listed by &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;: this is due to the fact that SLURM updates its internal data (used by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;) every few minutes, while the output of &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt; is updated much more frequently.&lt;br /&gt;
&lt;br /&gt;
=== How FairShare works ===&lt;br /&gt;
In Mufasa 2.0, the FairShare of each user is computed according to these rules:&lt;br /&gt;
# It considers the quantity used by (current and past) user jobs of:&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;CPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of CPUs used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;RAM&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the amount of RAM used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;GPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of GPUs used&lt;br /&gt;
#** note: 40 GB GPUs have double the impact on FairShare than 20 GB GPUs&lt;br /&gt;
# The impact on FairShare of any use of resources is &amp;#039;&amp;#039;&amp;#039;proportional to its duration&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
#* example: using 32 GB of RAM for 48h has the same impact of using 64 GB of RAM for 24h&lt;br /&gt;
# FairShare has a &amp;quot;fading memory&amp;quot;: i.e., resource use has less and less impact on FairShare the farther in the past it is&lt;br /&gt;
#* consequence: over time, the &amp;quot;history&amp;quot; of a user gets forgotten by FairShare&lt;br /&gt;
&lt;br /&gt;
== How to maximise the priority of your jobs ==&lt;br /&gt;
&lt;br /&gt;
Every time you run a SLURM job, follow these guidelines:&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Choose the less powerful QOS compatible with the needs of your job&lt;br /&gt;
:: QOSes with access to less resources have higher base priority.&lt;br /&gt;
&lt;br /&gt;
; Only request CPUs and RAM that your job really needs&lt;br /&gt;
:: Usually, code exploits multiple CPUs only if specifically designed to do so: if unsure, only ask for 1 CPU.&lt;br /&gt;
:: The fewer resources your job asks for, the less it will wait before they become available.&lt;br /&gt;
:: The more CPUs your job asks for, the stronger the penalty to its priority.&lt;br /&gt;
:: Asking for fewer CPUs and/or less RAM improves your FairShare.&lt;br /&gt;
&lt;br /&gt;
; Do not request more time than your jobs needs to complete&lt;br /&gt;
:: Make a worst-case estimate and only ask for that duration.&lt;br /&gt;
:: Asking for less of Mufasa&amp;#039;s time improves your FairShare.&lt;br /&gt;
&lt;br /&gt;
; Debug and test code using less powerful QOSes before running it with more powerful QOSes&lt;br /&gt;
:: Your test jobs will get a higher priority and your FairShare will improve.&lt;br /&gt;
&lt;br /&gt;
; Cancel jobs when you don&amp;#039;t need them anymore&lt;br /&gt;
:: Always use [[User_Jobs#Cancelling_a_job_with_scancel|scancel]] to delete completed (or crashed) jobs: your Fairshare will improve.&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Protip: it&amp;#039;s a good idea to [[User Jobs#Looking for unused GPUs|check for unused GPUs]] before choosing what to request. Requesting a GPU that is currently idle will help your job get executed sooner.&lt;br /&gt;
&lt;br /&gt;
= SLURM partitions =&lt;br /&gt;
&lt;br /&gt;
Partitions are another mechanism provided by SLURM to create different levels of access to system resources. Since in Mufasa 2.0 access to resources is controlled via [[#SLURM Quality of Service (QOS)|QOSes]], partitions are not very relevant. (This is one of the main differences between Mufasa 1.0 and Mufasa 2.0.)&lt;br /&gt;
&lt;br /&gt;
Note, however, that the default values for some features of SLURM jobs (e.g., duration) are [[#Default values|set by the partition]].&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, there is a single SLURM partition, called &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt;, and all jobs run on it. The state of &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt; can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sinfo -o &amp;quot;%10P %5a %9T %11L %10l&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PARTITION  AVAIL STATE     DEFAULTTIME TIMELIMIT &lt;br /&gt;
jobs*      up    idle      1:00:00     3-00:00:00&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where columns correspond to the following information:&lt;br /&gt;
&lt;br /&gt;
:; PARTITION&lt;br /&gt;
:: name of the partition; the asterisks indicates that it&amp;#039;s the default one&lt;br /&gt;
&lt;br /&gt;
:; AVAIL&lt;br /&gt;
:: state/availability of the partition: see [[#Partition availability|below]]&lt;br /&gt;
&lt;br /&gt;
:; STATE&lt;br /&gt;
:: state (using [https://slurm.schedmd.com/sinfo.html#SECTION_NODE-STATE-CODES these codes])&lt;br /&gt;
:: typical values are &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;mixed&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that some of the resources are busy executing jobs while other are idle, and &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;allocated&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that all of the resources are in use&lt;br /&gt;
&lt;br /&gt;
:; DEFAULTTIME&lt;br /&gt;
:: default runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:; TIMELIMIT&lt;br /&gt;
:: maximum runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
The asterisk at the end of the partition name indicates the default partition, i.e. the one on which jobs which do not ask for a specific partition are run.&lt;br /&gt;
&lt;br /&gt;
Command &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; does not tell you about the &amp;#039;&amp;#039;jobs&amp;#039;&amp;#039; submitted to a partition. This information is obtained, instead, with [[User Jobs#Inspecting jobs with squeue|command &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;]].&lt;br /&gt;
&lt;br /&gt;
== Partition availability ==&lt;br /&gt;
&lt;br /&gt;
The most important information that &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; provides is the &amp;#039;&amp;#039;&amp;#039;availability&amp;#039;&amp;#039;&amp;#039; (also called &amp;#039;&amp;#039;&amp;#039;state&amp;#039;&amp;#039;&amp;#039;) of partitions. This is shown in column &amp;quot;AVAIL&amp;quot;. Possible partition states are:&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is available&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed as soon as resources allow&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is in the process of becoming unavailable (i.e., of entering the &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; state: see below)&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is unavailable&lt;br /&gt;
:: There are no running jobs&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
When a partition goes from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; no harm is done to running jobs. In a normally functioning SLURM system, the passage from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; happens only when no jobs are running on the partition. If (e.g., due to a malfunction) the passage happens with jobs still running, they get killed.&lt;br /&gt;
&lt;br /&gt;
A partition in state &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; requires intervention by a [[Roles|Job Administrator]] to be restored to &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
== Default values ==&lt;br /&gt;
&lt;br /&gt;
The features of SLURM partitions, including the &amp;#039;&amp;#039;&amp;#039;default values&amp;#039;&amp;#039;&amp;#039; which are applied to jobs that do not make explicit requests, can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scontrol show partition&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PartitionName=jobs&lt;br /&gt;
   AllowGroups=ALL AllowAccounts=ALL AllowQos=nogpu,gpulight,gpu,gpuwide,gpuheavy-20,gpuheavy-40&lt;br /&gt;
   AllocNodes=ALL Default=YES QoS=N/A&lt;br /&gt;
   DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO&lt;br /&gt;
   MaxNodes=UNLIMITED MaxTime=3-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED&lt;br /&gt;
   Nodes=gn01&lt;br /&gt;
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO&lt;br /&gt;
   OverTimeLimit=NONE PreemptMode=OFF&lt;br /&gt;
   State=UP TotalCPUs=48 TotalNodes=1 SelectTypeParameters=NONE&lt;br /&gt;
   JobDefaults=(null)&lt;br /&gt;
   DefMemPerNode=4096 MaxMemPerNode=UNLIMITED&lt;br /&gt;
   TRES=cpu=48,mem=1011435M,node=1,billing=49,gres/gpu=13,gres/gpu:3g.20gb=5,gres/gpu:40gb=3,gres/gpu:4g.20gb=5&lt;br /&gt;
   TRESBillingWeights=cpu=1.0,gres/gpu:3g.20gb=6.0,gres/gpu:4g.20gb=6.0,gres/gpu:40gb=6.0,mem=0.05g&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In the output of &amp;lt;code&amp;gt;scontrol show partition&amp;lt;/code&amp;gt; the most relevant default values for Mufasa users are:&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefaultTime&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default execution time assigned to a job run on the partition (e.g., 1 hour)&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefMemPerNode&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default amount of RAM assigned to a job run on the partition (e.g., 4GB)&lt;br /&gt;
&lt;br /&gt;
= System resources subjected to limitations =&lt;br /&gt;
&lt;br /&gt;
In systems based on SLURM like Mufasa, &amp;#039;&amp;#039;&amp;#039;TRES (Trackable RESources)&amp;#039;&amp;#039;&amp;#039; are (from [https://slurm.schedmd.com/tres.html SLURM&amp;#039;s documentation] &amp;quot;&amp;#039;&amp;#039;resources that can be tracked for usage or used to enforce limits against.&amp;#039;&amp;#039;&amp;quot;&lt;br /&gt;
&lt;br /&gt;
TRES include CPUs, RAM and &amp;#039;&amp;#039;&amp;#039;GRES&amp;#039;&amp;#039;&amp;#039;. The last term stands for &amp;#039;&amp;#039;Generic RESources&amp;#039;&amp;#039; that a job may need for its execution. In Mufasa, the only &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; resources are the GPUs.&lt;br /&gt;
&lt;br /&gt;
== &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax ==&lt;br /&gt;
&lt;br /&gt;
To ask SLURM to assign GRES resources (i.e., GPUs) to a job, a special syntax must be used. Precisely, the name of each GPU resource takes the form&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:Name:Type&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
Considering the [[System#CPUs and GPUs|GPU complement of Mufasa]], &amp;lt;code&amp;gt;Type&amp;lt;/code&amp;gt; takes the following values:&lt;br /&gt;
&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:40gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 40 Gbytes of RAM&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:4g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 4 compute units&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 3 compute units&lt;br /&gt;
&lt;br /&gt;
So, for instance,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
identifies a resource corresponding to a GPU with 20 GB of RAM and 3 compute units.&lt;br /&gt;
&lt;br /&gt;
When asking for a GRES resource (e.g., in an &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command or an &amp;lt;code&amp;gt;SBATCH&amp;lt;/code&amp;gt; directive of an [[User Jobs#Using execution scripts to run jobs|execution script]]), the syntax required by SLURM is&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:&amp;lt;Type&amp;gt;:&amp;lt;Quantity&amp;gt;&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
where &amp;lt;code&amp;gt;Quantity&amp;lt;/code&amp;gt; is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of type &amp;lt;code&amp;gt;4g.20gb&amp;lt;/code&amp;gt; the syntax is&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:4g.20gb:2&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SLURM&amp;#039;s &amp;#039;&amp;#039;generic resources&amp;#039;&amp;#039; are defined in &amp;lt;code&amp;gt;/etc/slurm/gres.conf&amp;lt;/code&amp;gt;. In order to make GPUs available to SLURM&amp;#039;s &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; management, Mufasa makes use of Nvidia&amp;#039;s [https://developer.nvidia.com/nvidia-management-library-nvml NVML library]. For additional information see [https://slurm.schedmd.com/gres.html SLURM&amp;#039;s documentation].&lt;/div&gt;</summary>
		<author><name>GiulioFontana</name></author>
	</entry>
	<entry>
		<id>https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2603</id>
		<title>SLURM</title>
		<link rel="alternate" type="text/html" href="https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2603"/>
		<updated>2026-11-19T08:57:03Z</updated>

		<summary type="html">&lt;p&gt;GiulioFontana: /* Job priority */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page presents the features of SLURM that are most relevant to Mufasa&amp;#039;s [[Roles|Job Users]]. Job Users can submit jobs for execution, cancel their own jobs, and see other users&amp;#039; jobs (but not intervene on them).&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa &amp;#039;&amp;#039;&amp;#039;must use SLURM&amp;#039;&amp;#039;&amp;#039; to run resource-heavy processes, i.e. computing jobs that require one or more of the following:&lt;br /&gt;
* GPUs&lt;br /&gt;
* multiple CPUs&lt;br /&gt;
* powerful CPUs&lt;br /&gt;
* a significant amount of RAM&lt;br /&gt;
&lt;br /&gt;
In fact, only processes run via SLURM have access to all the resources of Mufasa. Processes run outside SLURM are executed by the [[System#Login server|login server]] virtual machine, which has minimal resources and no GPUs. Using SLURM is therefore the only way to execute resource-heavy jobs on Mufasa (this is a key difference between Mufasa 1.0 and Mufasa 2.0).&lt;br /&gt;
&lt;br /&gt;
= SLURM in a nutshell =&lt;br /&gt;
&lt;br /&gt;
Computation jobs on Mufasa needs to be launched via [[System#The SLURM job scheduling system|SLURM]]. SLURM provides jobs with access to the [[#System resources subjected to limitations|physical resources]] of Mufasa, such as CPUs, GPUs and RAM. Thanks to SLURM, processing jobs share system resources, optimising their occupation and availability. &lt;br /&gt;
&lt;br /&gt;
When a user runs a job, the job does not get executed immediately and is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039;. SLURM executes jobs according to their order in the queue: the top job in the queue gets executed as soon as the necessary resources are available, while jobs lower in the queue wait longer. The position of a job in the queue is due to the &amp;#039;&amp;#039;&amp;#039;[[#Job priority|priority]]&amp;#039;&amp;#039;&amp;#039; assigned to it by SLURM, with higher-priority jobs closer to the top. As a general rule:&lt;br /&gt;
&lt;br /&gt;
;: &amp;#039;&amp;#039;&amp;#039;the greater the fraction of Mufasa&amp;#039;s overall resources that a job asks for, the lower the job&amp;#039;s priority will be&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
The priority mechanism is used to encourage users to use Mufasa&amp;#039;s resources (i.e.: GPUs, CPUs, RAM, execution time) in an effective and equitable manner. This page includes a [[#How_to_maximise_the_priority_of_your_jobs|chart explaining how to maximise the priority of your jobs]].&lt;br /&gt;
&lt;br /&gt;
The &amp;#039;&amp;#039;&amp;#039;time&amp;#039;&amp;#039;&amp;#039; available to a job for its execution is controlled by SLURM. When a user requests execution of a job, they must specify the duration of the time slot that the job needs. The job must complete its execution before the end of the requested time slot, because at the end of such slot it gets killed by SLURM.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0 access to system resources is managed via SLURM&amp;#039;s [[#SLURM Quality of Service (QOS)|Quality of Service (QOS)]] mechanism (Mufasa 1.0 used [[#SLURM_partitions|partitions]] instead). While [[User Jobs#Running jobs with SLURM|launching a processing job via SLURM]], the user must always specify the chosen QOS. QOSes differ in the set of resources that they provide access to because each of them is designed to fit a given type of job.&lt;br /&gt;
&lt;br /&gt;
Mufasa sets limits to the number of jobs by the same user. This page includes a [[#Limits on jobs by the same user|table summarising such limits]].&lt;br /&gt;
&lt;br /&gt;
= SLURM Quality of Service (QOS) =&lt;br /&gt;
&lt;br /&gt;
Through &amp;#039;&amp;#039;&amp;#039;Quality of Services&amp;#039;&amp;#039;&amp;#039; (&amp;#039;&amp;#039;&amp;#039;QOSes&amp;#039;&amp;#039;&amp;#039;), SLURM lets system configurators assign a name to a set of related constraints.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, QOSes are used to define different levels of access to the server&amp;#039;s resources. When [[User Jobs|executing a job with SLURM]], a user must always &amp;#039;&amp;#039;&amp;#039;specify the QOS&amp;#039;&amp;#039;&amp;#039; that their job will use: this choice, in turn, determines what resources the job is able to access and influences the [[#Job priority|priority]] of the job.&lt;br /&gt;
&lt;br /&gt;
Mufasa&amp;#039;s QOSes and their features can be inspected with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list qos format=name%-11,priority,MaxSubmit,maxwall,maxtres%-80,mintres%-18&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following (see below for the same data in table form):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
Name          Priority MaxSubmit     MaxWall MaxTRES                                                                          MinTRES            &lt;br /&gt;
----------- ---------- --------- ----------- -------------------------------------------------------------------------------- ------------------ &lt;br /&gt;
normal               0                                                                                                                           &lt;br /&gt;
nogpu                4         1  3-00:00:00 cpu=16,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=128G                    &lt;br /&gt;
gpuheavy-20          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=2,mem=128G             gres/gpu:4g.20gb=1 &lt;br /&gt;
gpuheavy-40          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=1,gres/gpu:4g.20gb=0,mem=128G             gres/gpu:40gb=1    &lt;br /&gt;
gpulight             8         1    12:00:00 cpu=2,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpu                  2         1  1-00:00:00 cpu=8,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpuwide              2         2  1-00:00:00 cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=1,mem=64G              gres/gpu:4g.20gb=1 &lt;br /&gt;
build               32         1    02:00:00 cpu=2,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=16G&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The columns of this output are the following:&lt;br /&gt;
&lt;br /&gt;
:; Name&lt;br /&gt;
:: name of the QOS&lt;br /&gt;
&lt;br /&gt;
:; Priority&lt;br /&gt;
:: priority tier associated to the QOS (higher value = higher priority): see [[#Job priority|Job priority]] for details&lt;br /&gt;
&lt;br /&gt;
:; MaxSubmit&lt;br /&gt;
:: maximum number of jobs from a single user that can be submitted to SLURM with this QOS; submitted jobs include both running and queued jobs&lt;br /&gt;
:: See [[#Limits on jobs by the same user|Limits on jobs by the same user]] for an overview of the limits on jobs set by Mufasa.&lt;br /&gt;
&lt;br /&gt;
:; MaxWall&lt;br /&gt;
:: maximum wall clock duration of the jobs using the QOS (after which they are killed by SLURM), in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
:: For some QOSes these are not set: it means that they are determined by the [[#SLURM partitions|partition]]. Partitions also define the [[#Default values|default duration]] of jobs.&lt;br /&gt;
&lt;br /&gt;
:; MaxTRES&lt;br /&gt;
:: amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) available to a job using the QOS, where&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;cpu=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of CPUs (i.e., processor cores) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of CPUs specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;gres/&amp;#039;&amp;#039;gpu:Type&amp;#039;&amp;#039;=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of GPUs of class &amp;lt;code&amp;gt;&amp;#039;&amp;#039;Type&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; (see [[User Jobs#gres syntax|&amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax]]) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; (for QOSes that allow access to GPUs) if not specified, the job cannot be launched&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;mem=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;G&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum amount of system RAM is &amp;#039;&amp;#039;K&amp;#039;&amp;#039; GBytes&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of RAM specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
&lt;br /&gt;
:; MinTRES&lt;br /&gt;
:: minimum amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) that a job using the QOS must request in order to actually get executed by SLURM.&lt;br /&gt;
:: (If your job does not actually need these resources, you&amp;#039;ve chosen the wrong QOS: you can use one with a higher priority.)&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; QOS is the default one, and exists only to ensure that users always specify a QOS when running a job. Since &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; has zero priority and no resources, a job run using this QOS would never be run.&lt;br /&gt;
&lt;br /&gt;
The information provided by the &amp;lt;code&amp;gt;sacctmgr list qos&amp;lt;/code&amp;gt; command above is summarised by the following table:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! QOS&lt;br /&gt;
! Priority&amp;lt;/br&amp;gt;tier&lt;br /&gt;
! Max&amp;lt;/br&amp;gt;Submit&lt;br /&gt;
! MaxWall&amp;lt;/br&amp;gt;[h]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;CPUs&lt;br /&gt;
! max RAM&amp;lt;/br&amp;gt;[GB]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;3g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;4g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;40GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | build&lt;br /&gt;
| 32&lt;br /&gt;
| 1&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 16&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpulight&lt;br /&gt;
| 8&lt;br /&gt;
| 1&lt;br /&gt;
| 12&lt;br /&gt;
| 2&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | nogpu&lt;br /&gt;
| 4&lt;br /&gt;
| 1&lt;br /&gt;
| 72&lt;br /&gt;
| 16&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpu&lt;br /&gt;
| 2&lt;br /&gt;
| 1&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuwide&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-20&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| 2 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-40&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
A key piece of information in the table above is the &amp;#039;&amp;#039;&amp;#039;priority tier&amp;#039;&amp;#039;&amp;#039; associated to each QOS. In Mufasa 2.0, priority tiers are used to encourage users to use the &amp;#039;&amp;#039;&amp;#039;least powerful QOS that is compatible with their needs&amp;#039;&amp;#039;&amp;#039;, where &amp;quot;powerful&amp;quot; means &amp;quot;rich with resources&amp;quot;. Less powerful QOSes increase the priority of the jobs that use them, so these jobs tend to be executed sooner. See [[#Job priority|Job priority]] for details about how priority affects the execution order of jobs in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
== The &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS ==&lt;br /&gt;
&lt;br /&gt;
This QOS is specifically designed to be used by Mufasa users to &amp;#039;&amp;#039;&amp;#039;quickly build [[System#Containers|container images]]&amp;#039;&amp;#039;&amp;#039;. Its associated priority tier is very high: this, combined with the fact that jobs using this QOS require few resources, means that such jobs usually get executed very soon.&lt;br /&gt;
&lt;br /&gt;
On the other hand, the limited resources and the lack of access to the GPUs make the &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS unsuitable for other tasks.&lt;br /&gt;
&lt;br /&gt;
See [[Singularity#Building Singularity images|Building Singularity images]] for directions about building Singularity container images.&lt;br /&gt;
&lt;br /&gt;
== Restricted QOSes ==&lt;br /&gt;
&lt;br /&gt;
In Mufasa, the most powerful QOSes are reserved to researchers (called &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users: these include academic personnel and Ph.D. students). M.Sc. students, called &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users, cannot use them while running jobs.&lt;br /&gt;
&lt;br /&gt;
See [[#research users and students users|below]] to understand how &amp;#039;&amp;#039;user categories&amp;#039;&amp;#039; work in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
= &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users =&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa belong to two &amp;#039;&amp;#039;&amp;#039;user categories&amp;#039;&amp;#039;&amp;#039;. Categories provide users with different access to  to Mufasa&amp;#039;s resources: the idea being to provide researchers with more access while still letting students benefit from the server&amp;#039;s capabilities.&lt;br /&gt;
&lt;br /&gt;
User categories in Mufasa are the following:&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. academic personnel and Ph.D. students&lt;br /&gt;
::: * can use all QOSes, including the the [[#Restricted QOSes|restricted ones]]&lt;br /&gt;
::: * their jobs have a higher &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a higher number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. M.Sc. students&lt;br /&gt;
::: * cannot use the [[#Restricted QOSes|restricted QOSes]]&lt;br /&gt;
::: * their jobs have a lower &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a lower number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
You can inspect the differences in priority and running jobs between &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association format=&amp;quot;account,priority,maxjobs&amp;quot; | grep -E &amp;#039;Account|research|students&amp;#039;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
   Account   Priority MaxJobs &lt;br /&gt;
  research          4       2 &lt;br /&gt;
  students          1       1&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
To know what limits apply to your own user, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association where user=$USER format=&amp;quot;user,priority,maxjobs,qos%-60&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
      User   Priority MaxJobs QOS                                                          &lt;br /&gt;
---------- ---------- ------- ------------------------------------------------------------ &lt;br /&gt;
    preali          4       2 build,gpu,gpuheavy-20,gpuheavy-40,gpulight,gpuwide,nogpu&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The list under &amp;quot;QOS&amp;quot; shows what QOSes your user is allowed to use when [[User Jobs|running jobs]]. &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users can use all of them, while &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users can only access a subset of them.&lt;br /&gt;
&lt;br /&gt;
= Limits on jobs by the same user =&lt;br /&gt;
&lt;br /&gt;
Mufasa enforces limits on the number of jobs from a single user. Such limits aim at preventing users from &amp;quot;hogging&amp;quot; system resources, and apply to:&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;submitted jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that the user asked SLURM to execute, each of which may currently be either running or queued&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;running jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that are currently in execution&lt;br /&gt;
&lt;br /&gt;
Limits on submitted jobs are set via [[#SLURM Quality of Service (QOS)|QOSes]], while limits on running jobs are set via [[#research users and students users|user category]].&lt;br /&gt;
&lt;br /&gt;
The following table summarises what limits exist:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! &lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;running&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;submitted&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | global limits&amp;lt;br/&amp;gt;(system-wide)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to limits set by individual QOSes (below)&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | limits for&amp;lt;br/&amp;gt;individual&amp;lt;/br&amp;gt;QOSes&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to global limits (above)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for the&amp;#039;&amp;#039; &amp;lt;code&amp;gt;gpuwide&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;QOS&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for each of the other QOSes&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
= Job priority =&lt;br /&gt;
&lt;br /&gt;
Once the execution of a job has been requested, the job is not run immediately: it is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039; by SLURM, together with all the other jobs awaiting execution. The job on top of the queue at any time is the first to be put into execution as soon as the resources it requires are available. &lt;br /&gt;
&lt;br /&gt;
The order of the items in the job queue depends on their &amp;#039;&amp;#039;&amp;#039;priority&amp;#039;&amp;#039;&amp;#039;. The job with the highest priority is put by SLURM on top of the queue, while all other jobs are put in the queue in order of descending priority.&lt;br /&gt;
&lt;br /&gt;
The goal of SLURM is to maximise resource availability: i.e., to ensure the shorter possible wait time before job execution. To achieve this goal, in Mufasa SLURM is configured to &amp;#039;&amp;#039;&amp;#039;encourage users not to ask for resources or execution time that their job doesn&amp;#039;t need&amp;#039;&amp;#039;&amp;#039;. This is done via the priority mechanism: the more resources and/or time a job requests, the lower its priority will be; and the later it will be executed.&lt;br /&gt;
&lt;br /&gt;
Priority management in Mufasa is designed to set up a &amp;#039;&amp;#039;&amp;#039;virtuous cycle&amp;#039;&amp;#039;&amp;#039; where users, by carefully choosing what to ask for, obtain two results:&lt;br /&gt;
* they ensure that their job is executed as soon as possible;&lt;br /&gt;
* they leave as much as possible of Mufasa&amp;#039;s resources free for other users&amp;#039;s jobs.&lt;br /&gt;
&lt;br /&gt;
[[#How to maximise the priority of your jobs|Here]] you find tips to maximise the priority of your jobs.&lt;br /&gt;
&lt;br /&gt;
== Elements determining job priority ==&lt;br /&gt;
In Mufasa, the priority of a job is computed by SLURM according to the following elements:&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#research users and students users|User category]]&amp;#039;&amp;#039;&amp;#039; (i.e., &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;)&lt;br /&gt;
::: Used to provide higher priority to jobs run by &amp;#039;&amp;#039;&amp;#039;research personnel&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#SLURM Quality of Service (QOS)|QOS]]&amp;#039;&amp;#039;&amp;#039; used by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;less resources&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Number of CPUs&amp;#039;&amp;#039;&amp;#039; requested by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;a lower number of CPUs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job Age&amp;#039;&amp;#039;&amp;#039;, i.e. the time that the job has been waiting in the queue&lt;br /&gt;
::: Used to provide higher priority to jobs which have been &amp;#039;&amp;#039;&amp;#039;queued for a longer time&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
::: (Periods when a queued job is not eligible for running because of the [[#Limits on jobs by the same user|limits on running jobs by the same user]] do not increase job age)&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;FairShare&amp;#039;&amp;#039;&amp;#039;, i.e. a factor computed by SLURM to balance use of the system by different users&lt;br /&gt;
::: Used to provide higher priority to jobs by users who &amp;#039;&amp;#039;&amp;#039;used Mufasa&amp;#039;s resources less than others&amp;#039;&amp;#039;&amp;#039; (see [[#How FairShare works|below]])&lt;br /&gt;
&lt;br /&gt;
You can understand how each of these elements contributes to the priority of the jobs waiting for execution (both yours and of other users) with command &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;. A tailored version of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; that only displays the parameters listed above is the following:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sprio -o &amp;quot;%6i %9u %.8Y %.10B %.5Q %.9T %.5A %.5F&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Here is an example of the output of the command above:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
JOBID  USER      PRIORITY      ASSOC   QOS      TRES   AGE FAIRS&lt;br /&gt;
5790   amoglia       3495       2500   375  cpu=-800   933   487&lt;br /&gt;
5791   qliu          3010       2500   375  cpu=-800   916    19&lt;br /&gt;
5817   mzoccola      2336        625   750  cpu=-800   495  1266&lt;br /&gt;
5822   fbottazzi     2235        625   750  cpu=-600   467   994&lt;br /&gt;
5837   lpassoni      2508       2500   750  cpu=-800     0    58&lt;br /&gt;
5838   bagostini     4437        625  3000  cpu=-200     0  1013&lt;br /&gt;
5839   flazzaro      1549        625   750  cpu=-800     0   974&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
In it, column PRIORITY provides the overall priority of each job. Each priority value is the sum of the five values to its right; in particular, column ASSOC corresponds to user category, while column TRES corresponds to the number of CPUs. Note that the contribution of TRES to priority is &amp;#039;&amp;#039;negative&amp;#039;&amp;#039;, i.e. it is a penalty; its value is proportional to the number of CPUs requested by the job.&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;Things to consider when using &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
* Its output is only relevant for jobs &amp;#039;&amp;#039;not yet in execution&amp;#039;&amp;#039;. For an already running job, the values shown by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; correspond to the priority that the job would get if evaluated now: however, this priority does not affect job execution in any way.&lt;br /&gt;
* Jobs listed by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; may temporarily differ from jobs listed by &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;: this is due to the fact that SLURM updates its internal data (used by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;) every few minutes, while the output of &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt; is updated much more frequently.&lt;br /&gt;
&lt;br /&gt;
=== How FairShare works ===&lt;br /&gt;
In Mufasa 2.0, the FairShare of each user is computed according to these rules:&lt;br /&gt;
# It considers the quantity used by (current and past) user jobs of:&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;CPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of CPUs used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;RAM&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the amount of RAM used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;GPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of GPUs used&lt;br /&gt;
#** note: 40 GB GPUs have double the impact on FairShare than 20 GB GPUs&lt;br /&gt;
# The impact on FairShare of any use of resources is &amp;#039;&amp;#039;&amp;#039;proportional to its duration&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
#* example: using 32 GB of RAM for 48h has the same impact of using 64 GB of RAM for 24h&lt;br /&gt;
# FairShare has a &amp;quot;fading memory&amp;quot;: i.e., resource use has less and less impact on FairShare the farther in the past it is&lt;br /&gt;
#* consequence: over time, the &amp;quot;history&amp;quot; of a user gets forgotten by FairShare&lt;br /&gt;
&lt;br /&gt;
== How to maximise the priority of your jobs ==&lt;br /&gt;
&lt;br /&gt;
Every time you run a SLURM job, follow these guidelines:&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Choose the less powerful QOS compatible with the needs of your job&lt;br /&gt;
:: QOSes with access to less resources have higher base priority.&lt;br /&gt;
&lt;br /&gt;
; Only request CPUs and RAM that your job really needs&lt;br /&gt;
:: Usually, code exploits multiple CPUs only if specifically designed to do so: if unsure, only ask for 1 CPU.&lt;br /&gt;
:: The fewer resources your job asks for, the less it will wait before they become available.&lt;br /&gt;
:: The more CPUs your job asks for, the stronger the penalty to its priority.&lt;br /&gt;
:: Asking for fewer CPUs and/or less RAM improves your FairShare.&lt;br /&gt;
&lt;br /&gt;
; Do not request more time than your jobs needs to complete&lt;br /&gt;
:: Make a worst-case estimate and only ask for that duration.&lt;br /&gt;
:: Asking for less of Mufasa&amp;#039;s time improves your FairShare.&lt;br /&gt;
&lt;br /&gt;
; Debug and test code using less powerful QOSes before running it with more powerful QOSes&lt;br /&gt;
:: Your test jobs will get a higher priority and your FairShare will improve.&lt;br /&gt;
&lt;br /&gt;
; Cancel jobs when you don&amp;#039;t need them anymore&lt;br /&gt;
:: Always use [[User_Jobs#Cancelling_a_job_with_scancel|scancel]] to delete completed (or crashed) jobs: your Fairshare will improve.&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Protip: it&amp;#039;s a good idea to [[User Jobs#Looking for unused GPUs|check for unused GPUs]] before choosing what to request. Requesting a GPU that is currently idle will help your job get executed sooner.&lt;br /&gt;
&lt;br /&gt;
= SLURM partitions =&lt;br /&gt;
&lt;br /&gt;
Partitions are another mechanism provided by SLURM to create different levels of access to system resources. Since in Mufasa 2.0 access to resources is controlled via [[#SLURM Quality of Service (QOS)|QOSes]], partitions are not very relevant. (This is one of the main differences between Mufasa 1.0 and Mufasa 2.0.)&lt;br /&gt;
&lt;br /&gt;
Note, however, that the default values for some features of SLURM jobs (e.g., duration) are [[#Default values|set by the partition]].&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, there is a single SLURM partition, called &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt;, and all jobs run on it. The state of &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt; can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sinfo -o &amp;quot;%10P %5a %9T %11L %10l&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PARTITION  AVAIL STATE     DEFAULTTIME TIMELIMIT &lt;br /&gt;
jobs*      up    idle      1:00:00     3-00:00:00&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where columns correspond to the following information:&lt;br /&gt;
&lt;br /&gt;
:; PARTITION&lt;br /&gt;
:: name of the partition; the asterisks indicates that it&amp;#039;s the default one&lt;br /&gt;
&lt;br /&gt;
:; AVAIL&lt;br /&gt;
:: state/availability of the partition: see [[#Partition availability|below]]&lt;br /&gt;
&lt;br /&gt;
:; STATE&lt;br /&gt;
:: state (using [https://slurm.schedmd.com/sinfo.html#SECTION_NODE-STATE-CODES these codes])&lt;br /&gt;
:: typical values are &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;mixed&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that some of the resources are busy executing jobs while other are idle, and &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;allocated&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that all of the resources are in use&lt;br /&gt;
&lt;br /&gt;
:; DEFAULTTIME&lt;br /&gt;
:: default runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:; TIMELIMIT&lt;br /&gt;
:: maximum runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
The asterisk at the end of the partition name indicates the default partition, i.e. the one on which jobs which do not ask for a specific partition are run.&lt;br /&gt;
&lt;br /&gt;
Command &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; does not tell you about the &amp;#039;&amp;#039;jobs&amp;#039;&amp;#039; submitted to a partition. This information is obtained, instead, with [[User Jobs#Inspecting jobs with squeue|command &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;]].&lt;br /&gt;
&lt;br /&gt;
== Partition availability ==&lt;br /&gt;
&lt;br /&gt;
The most important information that &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; provides is the &amp;#039;&amp;#039;&amp;#039;availability&amp;#039;&amp;#039;&amp;#039; (also called &amp;#039;&amp;#039;&amp;#039;state&amp;#039;&amp;#039;&amp;#039;) of partitions. This is shown in column &amp;quot;AVAIL&amp;quot;. Possible partition states are:&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is available&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed as soon as resources allow&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is in the process of becoming unavailable (i.e., of entering the &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; state: see below)&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is unavailable&lt;br /&gt;
:: There are no running jobs&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
When a partition goes from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; no harm is done to running jobs. In a normally functioning SLURM system, the passage from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; happens only when no jobs are running on the partition. If (e.g., due to a malfunction) the passage happens with jobs still running, they get killed.&lt;br /&gt;
&lt;br /&gt;
A partition in state &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; requires intervention by a [[Roles|Job Administrator]] to be restored to &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
== Default values ==&lt;br /&gt;
&lt;br /&gt;
The features of SLURM partitions, including the &amp;#039;&amp;#039;&amp;#039;default values&amp;#039;&amp;#039;&amp;#039; which are applied to jobs that do not make explicit requests, can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scontrol show partition&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PartitionName=jobs&lt;br /&gt;
   AllowGroups=ALL AllowAccounts=ALL AllowQos=nogpu,gpulight,gpu,gpuwide,gpuheavy-20,gpuheavy-40&lt;br /&gt;
   AllocNodes=ALL Default=YES QoS=N/A&lt;br /&gt;
   DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO&lt;br /&gt;
   MaxNodes=UNLIMITED MaxTime=3-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED&lt;br /&gt;
   Nodes=gn01&lt;br /&gt;
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO&lt;br /&gt;
   OverTimeLimit=NONE PreemptMode=OFF&lt;br /&gt;
   State=UP TotalCPUs=48 TotalNodes=1 SelectTypeParameters=NONE&lt;br /&gt;
   JobDefaults=(null)&lt;br /&gt;
   DefMemPerNode=4096 MaxMemPerNode=UNLIMITED&lt;br /&gt;
   TRES=cpu=48,mem=1011435M,node=1,billing=49,gres/gpu=13,gres/gpu:3g.20gb=5,gres/gpu:40gb=3,gres/gpu:4g.20gb=5&lt;br /&gt;
   TRESBillingWeights=cpu=1.0,gres/gpu:3g.20gb=6.0,gres/gpu:4g.20gb=6.0,gres/gpu:40gb=6.0,mem=0.05g&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In the output of &amp;lt;code&amp;gt;scontrol show partition&amp;lt;/code&amp;gt; the most relevant default values for Mufasa users are:&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefaultTime&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default execution time assigned to a job run on the partition (e.g., 1 hour)&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefMemPerNode&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default amount of RAM assigned to a job run on the partition (e.g., 4GB)&lt;br /&gt;
&lt;br /&gt;
= System resources subjected to limitations =&lt;br /&gt;
&lt;br /&gt;
In systems based on SLURM like Mufasa, &amp;#039;&amp;#039;&amp;#039;TRES (Trackable RESources)&amp;#039;&amp;#039;&amp;#039; are (from [https://slurm.schedmd.com/tres.html SLURM&amp;#039;s documentation] &amp;quot;&amp;#039;&amp;#039;resources that can be tracked for usage or used to enforce limits against.&amp;#039;&amp;#039;&amp;quot;&lt;br /&gt;
&lt;br /&gt;
TRES include CPUs, RAM and &amp;#039;&amp;#039;&amp;#039;GRES&amp;#039;&amp;#039;&amp;#039;. The last term stands for &amp;#039;&amp;#039;Generic RESources&amp;#039;&amp;#039; that a job may need for its execution. In Mufasa, the only &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; resources are the GPUs.&lt;br /&gt;
&lt;br /&gt;
== &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax ==&lt;br /&gt;
&lt;br /&gt;
To ask SLURM to assign GRES resources (i.e., GPUs) to a job, a special syntax must be used. Precisely, the name of each GPU resource takes the form&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:Name:Type&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
Considering the [[System#CPUs and GPUs|GPU complement of Mufasa]], &amp;lt;code&amp;gt;Type&amp;lt;/code&amp;gt; takes the following values:&lt;br /&gt;
&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:40gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 40 Gbytes of RAM&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:4g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 4 compute units&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 3 compute units&lt;br /&gt;
&lt;br /&gt;
So, for instance,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
identifies a resource corresponding to a GPU with 20 GB of RAM and 3 compute units.&lt;br /&gt;
&lt;br /&gt;
When asking for a GRES resource (e.g., in an &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command or an &amp;lt;code&amp;gt;SBATCH&amp;lt;/code&amp;gt; directive of an [[User Jobs#Using execution scripts to run jobs|execution script]]), the syntax required by SLURM is&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:&amp;lt;Type&amp;gt;:&amp;lt;Quantity&amp;gt;&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
where &amp;lt;code&amp;gt;Quantity&amp;lt;/code&amp;gt; is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of type &amp;lt;code&amp;gt;4g.20gb&amp;lt;/code&amp;gt; the syntax is&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:4g.20gb:2&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SLURM&amp;#039;s &amp;#039;&amp;#039;generic resources&amp;#039;&amp;#039; are defined in &amp;lt;code&amp;gt;/etc/slurm/gres.conf&amp;lt;/code&amp;gt;. In order to make GPUs available to SLURM&amp;#039;s &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; management, Mufasa makes use of Nvidia&amp;#039;s [https://developer.nvidia.com/nvidia-management-library-nvml NVML library]. For additional information see [https://slurm.schedmd.com/gres.html SLURM&amp;#039;s documentation].&lt;/div&gt;</summary>
		<author><name>GiulioFontana</name></author>
	</entry>
	<entry>
		<id>https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2602</id>
		<title>SLURM</title>
		<link rel="alternate" type="text/html" href="https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2602"/>
		<updated>2026-11-19T08:56:46Z</updated>

		<summary type="html">&lt;p&gt;GiulioFontana: /* Job priority */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page presents the features of SLURM that are most relevant to Mufasa&amp;#039;s [[Roles|Job Users]]. Job Users can submit jobs for execution, cancel their own jobs, and see other users&amp;#039; jobs (but not intervene on them).&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa &amp;#039;&amp;#039;&amp;#039;must use SLURM&amp;#039;&amp;#039;&amp;#039; to run resource-heavy processes, i.e. computing jobs that require one or more of the following:&lt;br /&gt;
* GPUs&lt;br /&gt;
* multiple CPUs&lt;br /&gt;
* powerful CPUs&lt;br /&gt;
* a significant amount of RAM&lt;br /&gt;
&lt;br /&gt;
In fact, only processes run via SLURM have access to all the resources of Mufasa. Processes run outside SLURM are executed by the [[System#Login server|login server]] virtual machine, which has minimal resources and no GPUs. Using SLURM is therefore the only way to execute resource-heavy jobs on Mufasa (this is a key difference between Mufasa 1.0 and Mufasa 2.0).&lt;br /&gt;
&lt;br /&gt;
= SLURM in a nutshell =&lt;br /&gt;
&lt;br /&gt;
Computation jobs on Mufasa needs to be launched via [[System#The SLURM job scheduling system|SLURM]]. SLURM provides jobs with access to the [[#System resources subjected to limitations|physical resources]] of Mufasa, such as CPUs, GPUs and RAM. Thanks to SLURM, processing jobs share system resources, optimising their occupation and availability. &lt;br /&gt;
&lt;br /&gt;
When a user runs a job, the job does not get executed immediately and is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039;. SLURM executes jobs according to their order in the queue: the top job in the queue gets executed as soon as the necessary resources are available, while jobs lower in the queue wait longer. The position of a job in the queue is due to the &amp;#039;&amp;#039;&amp;#039;[[#Job priority|priority]]&amp;#039;&amp;#039;&amp;#039; assigned to it by SLURM, with higher-priority jobs closer to the top. As a general rule:&lt;br /&gt;
&lt;br /&gt;
;: &amp;#039;&amp;#039;&amp;#039;the greater the fraction of Mufasa&amp;#039;s overall resources that a job asks for, the lower the job&amp;#039;s priority will be&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
The priority mechanism is used to encourage users to use Mufasa&amp;#039;s resources (i.e.: GPUs, CPUs, RAM, execution time) in an effective and equitable manner. This page includes a [[#How_to_maximise_the_priority_of_your_jobs|chart explaining how to maximise the priority of your jobs]].&lt;br /&gt;
&lt;br /&gt;
The &amp;#039;&amp;#039;&amp;#039;time&amp;#039;&amp;#039;&amp;#039; available to a job for its execution is controlled by SLURM. When a user requests execution of a job, they must specify the duration of the time slot that the job needs. The job must complete its execution before the end of the requested time slot, because at the end of such slot it gets killed by SLURM.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0 access to system resources is managed via SLURM&amp;#039;s [[#SLURM Quality of Service (QOS)|Quality of Service (QOS)]] mechanism (Mufasa 1.0 used [[#SLURM_partitions|partitions]] instead). While [[User Jobs#Running jobs with SLURM|launching a processing job via SLURM]], the user must always specify the chosen QOS. QOSes differ in the set of resources that they provide access to because each of them is designed to fit a given type of job.&lt;br /&gt;
&lt;br /&gt;
Mufasa sets limits to the number of jobs by the same user. This page includes a [[#Limits on jobs by the same user|table summarising such limits]].&lt;br /&gt;
&lt;br /&gt;
= SLURM Quality of Service (QOS) =&lt;br /&gt;
&lt;br /&gt;
Through &amp;#039;&amp;#039;&amp;#039;Quality of Services&amp;#039;&amp;#039;&amp;#039; (&amp;#039;&amp;#039;&amp;#039;QOSes&amp;#039;&amp;#039;&amp;#039;), SLURM lets system configurators assign a name to a set of related constraints.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, QOSes are used to define different levels of access to the server&amp;#039;s resources. When [[User Jobs|executing a job with SLURM]], a user must always &amp;#039;&amp;#039;&amp;#039;specify the QOS&amp;#039;&amp;#039;&amp;#039; that their job will use: this choice, in turn, determines what resources the job is able to access and influences the [[#Job priority|priority]] of the job.&lt;br /&gt;
&lt;br /&gt;
Mufasa&amp;#039;s QOSes and their features can be inspected with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list qos format=name%-11,priority,MaxSubmit,maxwall,maxtres%-80,mintres%-18&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following (see below for the same data in table form):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
Name          Priority MaxSubmit     MaxWall MaxTRES                                                                          MinTRES            &lt;br /&gt;
----------- ---------- --------- ----------- -------------------------------------------------------------------------------- ------------------ &lt;br /&gt;
normal               0                                                                                                                           &lt;br /&gt;
nogpu                4         1  3-00:00:00 cpu=16,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=128G                    &lt;br /&gt;
gpuheavy-20          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=2,mem=128G             gres/gpu:4g.20gb=1 &lt;br /&gt;
gpuheavy-40          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=1,gres/gpu:4g.20gb=0,mem=128G             gres/gpu:40gb=1    &lt;br /&gt;
gpulight             8         1    12:00:00 cpu=2,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpu                  2         1  1-00:00:00 cpu=8,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpuwide              2         2  1-00:00:00 cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=1,mem=64G              gres/gpu:4g.20gb=1 &lt;br /&gt;
build               32         1    02:00:00 cpu=2,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=16G&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The columns of this output are the following:&lt;br /&gt;
&lt;br /&gt;
:; Name&lt;br /&gt;
:: name of the QOS&lt;br /&gt;
&lt;br /&gt;
:; Priority&lt;br /&gt;
:: priority tier associated to the QOS (higher value = higher priority): see [[#Job priority|Job priority]] for details&lt;br /&gt;
&lt;br /&gt;
:; MaxSubmit&lt;br /&gt;
:: maximum number of jobs from a single user that can be submitted to SLURM with this QOS; submitted jobs include both running and queued jobs&lt;br /&gt;
:: See [[#Limits on jobs by the same user|Limits on jobs by the same user]] for an overview of the limits on jobs set by Mufasa.&lt;br /&gt;
&lt;br /&gt;
:; MaxWall&lt;br /&gt;
:: maximum wall clock duration of the jobs using the QOS (after which they are killed by SLURM), in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
:: For some QOSes these are not set: it means that they are determined by the [[#SLURM partitions|partition]]. Partitions also define the [[#Default values|default duration]] of jobs.&lt;br /&gt;
&lt;br /&gt;
:; MaxTRES&lt;br /&gt;
:: amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) available to a job using the QOS, where&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;cpu=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of CPUs (i.e., processor cores) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of CPUs specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;gres/&amp;#039;&amp;#039;gpu:Type&amp;#039;&amp;#039;=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of GPUs of class &amp;lt;code&amp;gt;&amp;#039;&amp;#039;Type&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; (see [[User Jobs#gres syntax|&amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax]]) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; (for QOSes that allow access to GPUs) if not specified, the job cannot be launched&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;mem=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;G&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum amount of system RAM is &amp;#039;&amp;#039;K&amp;#039;&amp;#039; GBytes&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of RAM specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
&lt;br /&gt;
:; MinTRES&lt;br /&gt;
:: minimum amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) that a job using the QOS must request in order to actually get executed by SLURM.&lt;br /&gt;
:: (If your job does not actually need these resources, you&amp;#039;ve chosen the wrong QOS: you can use one with a higher priority.)&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; QOS is the default one, and exists only to ensure that users always specify a QOS when running a job. Since &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; has zero priority and no resources, a job run using this QOS would never be run.&lt;br /&gt;
&lt;br /&gt;
The information provided by the &amp;lt;code&amp;gt;sacctmgr list qos&amp;lt;/code&amp;gt; command above is summarised by the following table:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! QOS&lt;br /&gt;
! Priority&amp;lt;/br&amp;gt;tier&lt;br /&gt;
! Max&amp;lt;/br&amp;gt;Submit&lt;br /&gt;
! MaxWall&amp;lt;/br&amp;gt;[h]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;CPUs&lt;br /&gt;
! max RAM&amp;lt;/br&amp;gt;[GB]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;3g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;4g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;40GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | build&lt;br /&gt;
| 32&lt;br /&gt;
| 1&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 16&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpulight&lt;br /&gt;
| 8&lt;br /&gt;
| 1&lt;br /&gt;
| 12&lt;br /&gt;
| 2&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | nogpu&lt;br /&gt;
| 4&lt;br /&gt;
| 1&lt;br /&gt;
| 72&lt;br /&gt;
| 16&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpu&lt;br /&gt;
| 2&lt;br /&gt;
| 1&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuwide&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-20&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| 2 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-40&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
A key piece of information in the table above is the &amp;#039;&amp;#039;&amp;#039;priority tier&amp;#039;&amp;#039;&amp;#039; associated to each QOS. In Mufasa 2.0, priority tiers are used to encourage users to use the &amp;#039;&amp;#039;&amp;#039;least powerful QOS that is compatible with their needs&amp;#039;&amp;#039;&amp;#039;, where &amp;quot;powerful&amp;quot; means &amp;quot;rich with resources&amp;quot;. Less powerful QOSes increase the priority of the jobs that use them, so these jobs tend to be executed sooner. See [[#Job priority|Job priority]] for details about how priority affects the execution order of jobs in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
== The &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS ==&lt;br /&gt;
&lt;br /&gt;
This QOS is specifically designed to be used by Mufasa users to &amp;#039;&amp;#039;&amp;#039;quickly build [[System#Containers|container images]]&amp;#039;&amp;#039;&amp;#039;. Its associated priority tier is very high: this, combined with the fact that jobs using this QOS require few resources, means that such jobs usually get executed very soon.&lt;br /&gt;
&lt;br /&gt;
On the other hand, the limited resources and the lack of access to the GPUs make the &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS unsuitable for other tasks.&lt;br /&gt;
&lt;br /&gt;
See [[Singularity#Building Singularity images|Building Singularity images]] for directions about building Singularity container images.&lt;br /&gt;
&lt;br /&gt;
== Restricted QOSes ==&lt;br /&gt;
&lt;br /&gt;
In Mufasa, the most powerful QOSes are reserved to researchers (called &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users: these include academic personnel and Ph.D. students). M.Sc. students, called &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users, cannot use them while running jobs.&lt;br /&gt;
&lt;br /&gt;
See [[#research users and students users|below]] to understand how &amp;#039;&amp;#039;user categories&amp;#039;&amp;#039; work in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
= &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users =&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa belong to two &amp;#039;&amp;#039;&amp;#039;user categories&amp;#039;&amp;#039;&amp;#039;. Categories provide users with different access to  to Mufasa&amp;#039;s resources: the idea being to provide researchers with more access while still letting students benefit from the server&amp;#039;s capabilities.&lt;br /&gt;
&lt;br /&gt;
User categories in Mufasa are the following:&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. academic personnel and Ph.D. students&lt;br /&gt;
::: * can use all QOSes, including the the [[#Restricted QOSes|restricted ones]]&lt;br /&gt;
::: * their jobs have a higher &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a higher number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. M.Sc. students&lt;br /&gt;
::: * cannot use the [[#Restricted QOSes|restricted QOSes]]&lt;br /&gt;
::: * their jobs have a lower &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a lower number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
You can inspect the differences in priority and running jobs between &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association format=&amp;quot;account,priority,maxjobs&amp;quot; | grep -E &amp;#039;Account|research|students&amp;#039;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
   Account   Priority MaxJobs &lt;br /&gt;
  research          4       2 &lt;br /&gt;
  students          1       1&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
To know what limits apply to your own user, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association where user=$USER format=&amp;quot;user,priority,maxjobs,qos%-60&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
      User   Priority MaxJobs QOS                                                          &lt;br /&gt;
---------- ---------- ------- ------------------------------------------------------------ &lt;br /&gt;
    preali          4       2 build,gpu,gpuheavy-20,gpuheavy-40,gpulight,gpuwide,nogpu&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The list under &amp;quot;QOS&amp;quot; shows what QOSes your user is allowed to use when [[User Jobs|running jobs]]. &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users can use all of them, while &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users can only access a subset of them.&lt;br /&gt;
&lt;br /&gt;
= Limits on jobs by the same user =&lt;br /&gt;
&lt;br /&gt;
Mufasa enforces limits on the number of jobs from a single user. Such limits aim at preventing users from &amp;quot;hogging&amp;quot; system resources, and apply to:&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;submitted jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that the user asked SLURM to execute, each of which may currently be either running or queued&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;running jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that are currently in execution&lt;br /&gt;
&lt;br /&gt;
Limits on submitted jobs are set via [[#SLURM Quality of Service (QOS)|QOSes]], while limits on running jobs are set via [[#research users and students users|user category]].&lt;br /&gt;
&lt;br /&gt;
The following table summarises what limits exist:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! &lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;running&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;submitted&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | global limits&amp;lt;br/&amp;gt;(system-wide)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to limits set by individual QOSes (below)&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | limits for&amp;lt;br/&amp;gt;individual&amp;lt;/br&amp;gt;QOSes&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to global limits (above)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for the&amp;#039;&amp;#039; &amp;lt;code&amp;gt;gpuwide&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;QOS&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for each of the other QOSes&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
= Job priority =&lt;br /&gt;
&lt;br /&gt;
Once the execution of a job has been requested, the job is not run immediately: it is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039; by SLURM, together with all the other jobs awaiting execution. The job on top of the queue at any time is the first to be put into execution as soon as the resources it requires are available. &lt;br /&gt;
&lt;br /&gt;
The order of the items in the job queue depends on their &amp;#039;&amp;#039;&amp;#039;priority&amp;#039;&amp;#039;&amp;#039;. The job with the highest priority is put by SLURM on top of the queue, while all other jobs are put in the queue in order of descending priority.&lt;br /&gt;
&lt;br /&gt;
The goal of SLURM is to maximise resource availability: i.e., to ensure the shorter possible wait time before job execution. To achieve this goal, in Mufasa SLURM is configured to &amp;#039;&amp;#039;&amp;#039;encourage users not to ask for resources or execution time that their job doesn&amp;#039;t need&amp;#039;&amp;#039;&amp;#039;. This is done via the priority mechanism: the more resources and/or time a job requests, the lower its priority will be; and the later it will be executed.&lt;br /&gt;
&lt;br /&gt;
Priority management in Mufasa is designed to set up a &amp;#039;&amp;#039;&amp;#039;virtuous cycle&amp;#039;&amp;#039;&amp;#039; where users, by carefully choosing what to ask for, obtain two results:&lt;br /&gt;
* they ensure that their job is executed as soon as possible;&lt;br /&gt;
* they leave as much as possible of Mufasa&amp;#039;s resources free for other users&amp;#039;s jobs.&lt;br /&gt;
&lt;br /&gt;
[#How to maximise the priority of your jobs|Here] you find tips to maximise the priority of your jobs.&lt;br /&gt;
&lt;br /&gt;
== Elements determining job priority ==&lt;br /&gt;
In Mufasa, the priority of a job is computed by SLURM according to the following elements:&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#research users and students users|User category]]&amp;#039;&amp;#039;&amp;#039; (i.e., &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;)&lt;br /&gt;
::: Used to provide higher priority to jobs run by &amp;#039;&amp;#039;&amp;#039;research personnel&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#SLURM Quality of Service (QOS)|QOS]]&amp;#039;&amp;#039;&amp;#039; used by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;less resources&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Number of CPUs&amp;#039;&amp;#039;&amp;#039; requested by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;a lower number of CPUs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job Age&amp;#039;&amp;#039;&amp;#039;, i.e. the time that the job has been waiting in the queue&lt;br /&gt;
::: Used to provide higher priority to jobs which have been &amp;#039;&amp;#039;&amp;#039;queued for a longer time&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
::: (Periods when a queued job is not eligible for running because of the [[#Limits on jobs by the same user|limits on running jobs by the same user]] do not increase job age)&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;FairShare&amp;#039;&amp;#039;&amp;#039;, i.e. a factor computed by SLURM to balance use of the system by different users&lt;br /&gt;
::: Used to provide higher priority to jobs by users who &amp;#039;&amp;#039;&amp;#039;used Mufasa&amp;#039;s resources less than others&amp;#039;&amp;#039;&amp;#039; (see [[#How FairShare works|below]])&lt;br /&gt;
&lt;br /&gt;
You can understand how each of these elements contributes to the priority of the jobs waiting for execution (both yours and of other users) with command &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;. A tailored version of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; that only displays the parameters listed above is the following:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sprio -o &amp;quot;%6i %9u %.8Y %.10B %.5Q %.9T %.5A %.5F&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Here is an example of the output of the command above:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
JOBID  USER      PRIORITY      ASSOC   QOS      TRES   AGE FAIRS&lt;br /&gt;
5790   amoglia       3495       2500   375  cpu=-800   933   487&lt;br /&gt;
5791   qliu          3010       2500   375  cpu=-800   916    19&lt;br /&gt;
5817   mzoccola      2336        625   750  cpu=-800   495  1266&lt;br /&gt;
5822   fbottazzi     2235        625   750  cpu=-600   467   994&lt;br /&gt;
5837   lpassoni      2508       2500   750  cpu=-800     0    58&lt;br /&gt;
5838   bagostini     4437        625  3000  cpu=-200     0  1013&lt;br /&gt;
5839   flazzaro      1549        625   750  cpu=-800     0   974&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
In it, column PRIORITY provides the overall priority of each job. Each priority value is the sum of the five values to its right; in particular, column ASSOC corresponds to user category, while column TRES corresponds to the number of CPUs. Note that the contribution of TRES to priority is &amp;#039;&amp;#039;negative&amp;#039;&amp;#039;, i.e. it is a penalty; its value is proportional to the number of CPUs requested by the job.&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;Things to consider when using &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
* Its output is only relevant for jobs &amp;#039;&amp;#039;not yet in execution&amp;#039;&amp;#039;. For an already running job, the values shown by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; correspond to the priority that the job would get if evaluated now: however, this priority does not affect job execution in any way.&lt;br /&gt;
* Jobs listed by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; may temporarily differ from jobs listed by &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;: this is due to the fact that SLURM updates its internal data (used by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;) every few minutes, while the output of &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt; is updated much more frequently.&lt;br /&gt;
&lt;br /&gt;
=== How FairShare works ===&lt;br /&gt;
In Mufasa 2.0, the FairShare of each user is computed according to these rules:&lt;br /&gt;
# It considers the quantity used by (current and past) user jobs of:&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;CPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of CPUs used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;RAM&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the amount of RAM used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;GPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of GPUs used&lt;br /&gt;
#** note: 40 GB GPUs have double the impact on FairShare than 20 GB GPUs&lt;br /&gt;
# The impact on FairShare of any use of resources is &amp;#039;&amp;#039;&amp;#039;proportional to its duration&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
#* example: using 32 GB of RAM for 48h has the same impact of using 64 GB of RAM for 24h&lt;br /&gt;
# FairShare has a &amp;quot;fading memory&amp;quot;: i.e., resource use has less and less impact on FairShare the farther in the past it is&lt;br /&gt;
#* consequence: over time, the &amp;quot;history&amp;quot; of a user gets forgotten by FairShare&lt;br /&gt;
&lt;br /&gt;
== How to maximise the priority of your jobs ==&lt;br /&gt;
&lt;br /&gt;
Every time you run a SLURM job, follow these guidelines:&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Choose the less powerful QOS compatible with the needs of your job&lt;br /&gt;
:: QOSes with access to less resources have higher base priority.&lt;br /&gt;
&lt;br /&gt;
; Only request CPUs and RAM that your job really needs&lt;br /&gt;
:: Usually, code exploits multiple CPUs only if specifically designed to do so: if unsure, only ask for 1 CPU.&lt;br /&gt;
:: The fewer resources your job asks for, the less it will wait before they become available.&lt;br /&gt;
:: The more CPUs your job asks for, the stronger the penalty to its priority.&lt;br /&gt;
:: Asking for fewer CPUs and/or less RAM improves your FairShare.&lt;br /&gt;
&lt;br /&gt;
; Do not request more time than your jobs needs to complete&lt;br /&gt;
:: Make a worst-case estimate and only ask for that duration.&lt;br /&gt;
:: Asking for less of Mufasa&amp;#039;s time improves your FairShare.&lt;br /&gt;
&lt;br /&gt;
; Debug and test code using less powerful QOSes before running it with more powerful QOSes&lt;br /&gt;
:: Your test jobs will get a higher priority and your FairShare will improve.&lt;br /&gt;
&lt;br /&gt;
; Cancel jobs when you don&amp;#039;t need them anymore&lt;br /&gt;
:: Always use [[User_Jobs#Cancelling_a_job_with_scancel|scancel]] to delete completed (or crashed) jobs: your Fairshare will improve.&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Protip: it&amp;#039;s a good idea to [[User Jobs#Looking for unused GPUs|check for unused GPUs]] before choosing what to request. Requesting a GPU that is currently idle will help your job get executed sooner.&lt;br /&gt;
&lt;br /&gt;
= SLURM partitions =&lt;br /&gt;
&lt;br /&gt;
Partitions are another mechanism provided by SLURM to create different levels of access to system resources. Since in Mufasa 2.0 access to resources is controlled via [[#SLURM Quality of Service (QOS)|QOSes]], partitions are not very relevant. (This is one of the main differences between Mufasa 1.0 and Mufasa 2.0.)&lt;br /&gt;
&lt;br /&gt;
Note, however, that the default values for some features of SLURM jobs (e.g., duration) are [[#Default values|set by the partition]].&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, there is a single SLURM partition, called &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt;, and all jobs run on it. The state of &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt; can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sinfo -o &amp;quot;%10P %5a %9T %11L %10l&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PARTITION  AVAIL STATE     DEFAULTTIME TIMELIMIT &lt;br /&gt;
jobs*      up    idle      1:00:00     3-00:00:00&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where columns correspond to the following information:&lt;br /&gt;
&lt;br /&gt;
:; PARTITION&lt;br /&gt;
:: name of the partition; the asterisks indicates that it&amp;#039;s the default one&lt;br /&gt;
&lt;br /&gt;
:; AVAIL&lt;br /&gt;
:: state/availability of the partition: see [[#Partition availability|below]]&lt;br /&gt;
&lt;br /&gt;
:; STATE&lt;br /&gt;
:: state (using [https://slurm.schedmd.com/sinfo.html#SECTION_NODE-STATE-CODES these codes])&lt;br /&gt;
:: typical values are &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;mixed&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that some of the resources are busy executing jobs while other are idle, and &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;allocated&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that all of the resources are in use&lt;br /&gt;
&lt;br /&gt;
:; DEFAULTTIME&lt;br /&gt;
:: default runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:; TIMELIMIT&lt;br /&gt;
:: maximum runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
The asterisk at the end of the partition name indicates the default partition, i.e. the one on which jobs which do not ask for a specific partition are run.&lt;br /&gt;
&lt;br /&gt;
Command &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; does not tell you about the &amp;#039;&amp;#039;jobs&amp;#039;&amp;#039; submitted to a partition. This information is obtained, instead, with [[User Jobs#Inspecting jobs with squeue|command &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;]].&lt;br /&gt;
&lt;br /&gt;
== Partition availability ==&lt;br /&gt;
&lt;br /&gt;
The most important information that &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; provides is the &amp;#039;&amp;#039;&amp;#039;availability&amp;#039;&amp;#039;&amp;#039; (also called &amp;#039;&amp;#039;&amp;#039;state&amp;#039;&amp;#039;&amp;#039;) of partitions. This is shown in column &amp;quot;AVAIL&amp;quot;. Possible partition states are:&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is available&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed as soon as resources allow&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is in the process of becoming unavailable (i.e., of entering the &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; state: see below)&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is unavailable&lt;br /&gt;
:: There are no running jobs&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
When a partition goes from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; no harm is done to running jobs. In a normally functioning SLURM system, the passage from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; happens only when no jobs are running on the partition. If (e.g., due to a malfunction) the passage happens with jobs still running, they get killed.&lt;br /&gt;
&lt;br /&gt;
A partition in state &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; requires intervention by a [[Roles|Job Administrator]] to be restored to &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
== Default values ==&lt;br /&gt;
&lt;br /&gt;
The features of SLURM partitions, including the &amp;#039;&amp;#039;&amp;#039;default values&amp;#039;&amp;#039;&amp;#039; which are applied to jobs that do not make explicit requests, can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scontrol show partition&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PartitionName=jobs&lt;br /&gt;
   AllowGroups=ALL AllowAccounts=ALL AllowQos=nogpu,gpulight,gpu,gpuwide,gpuheavy-20,gpuheavy-40&lt;br /&gt;
   AllocNodes=ALL Default=YES QoS=N/A&lt;br /&gt;
   DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO&lt;br /&gt;
   MaxNodes=UNLIMITED MaxTime=3-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED&lt;br /&gt;
   Nodes=gn01&lt;br /&gt;
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO&lt;br /&gt;
   OverTimeLimit=NONE PreemptMode=OFF&lt;br /&gt;
   State=UP TotalCPUs=48 TotalNodes=1 SelectTypeParameters=NONE&lt;br /&gt;
   JobDefaults=(null)&lt;br /&gt;
   DefMemPerNode=4096 MaxMemPerNode=UNLIMITED&lt;br /&gt;
   TRES=cpu=48,mem=1011435M,node=1,billing=49,gres/gpu=13,gres/gpu:3g.20gb=5,gres/gpu:40gb=3,gres/gpu:4g.20gb=5&lt;br /&gt;
   TRESBillingWeights=cpu=1.0,gres/gpu:3g.20gb=6.0,gres/gpu:4g.20gb=6.0,gres/gpu:40gb=6.0,mem=0.05g&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In the output of &amp;lt;code&amp;gt;scontrol show partition&amp;lt;/code&amp;gt; the most relevant default values for Mufasa users are:&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefaultTime&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default execution time assigned to a job run on the partition (e.g., 1 hour)&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefMemPerNode&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default amount of RAM assigned to a job run on the partition (e.g., 4GB)&lt;br /&gt;
&lt;br /&gt;
= System resources subjected to limitations =&lt;br /&gt;
&lt;br /&gt;
In systems based on SLURM like Mufasa, &amp;#039;&amp;#039;&amp;#039;TRES (Trackable RESources)&amp;#039;&amp;#039;&amp;#039; are (from [https://slurm.schedmd.com/tres.html SLURM&amp;#039;s documentation] &amp;quot;&amp;#039;&amp;#039;resources that can be tracked for usage or used to enforce limits against.&amp;#039;&amp;#039;&amp;quot;&lt;br /&gt;
&lt;br /&gt;
TRES include CPUs, RAM and &amp;#039;&amp;#039;&amp;#039;GRES&amp;#039;&amp;#039;&amp;#039;. The last term stands for &amp;#039;&amp;#039;Generic RESources&amp;#039;&amp;#039; that a job may need for its execution. In Mufasa, the only &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; resources are the GPUs.&lt;br /&gt;
&lt;br /&gt;
== &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax ==&lt;br /&gt;
&lt;br /&gt;
To ask SLURM to assign GRES resources (i.e., GPUs) to a job, a special syntax must be used. Precisely, the name of each GPU resource takes the form&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:Name:Type&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
Considering the [[System#CPUs and GPUs|GPU complement of Mufasa]], &amp;lt;code&amp;gt;Type&amp;lt;/code&amp;gt; takes the following values:&lt;br /&gt;
&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:40gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 40 Gbytes of RAM&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:4g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 4 compute units&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 3 compute units&lt;br /&gt;
&lt;br /&gt;
So, for instance,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
identifies a resource corresponding to a GPU with 20 GB of RAM and 3 compute units.&lt;br /&gt;
&lt;br /&gt;
When asking for a GRES resource (e.g., in an &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command or an &amp;lt;code&amp;gt;SBATCH&amp;lt;/code&amp;gt; directive of an [[User Jobs#Using execution scripts to run jobs|execution script]]), the syntax required by SLURM is&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:&amp;lt;Type&amp;gt;:&amp;lt;Quantity&amp;gt;&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
where &amp;lt;code&amp;gt;Quantity&amp;lt;/code&amp;gt; is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of type &amp;lt;code&amp;gt;4g.20gb&amp;lt;/code&amp;gt; the syntax is&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:4g.20gb:2&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SLURM&amp;#039;s &amp;#039;&amp;#039;generic resources&amp;#039;&amp;#039; are defined in &amp;lt;code&amp;gt;/etc/slurm/gres.conf&amp;lt;/code&amp;gt;. In order to make GPUs available to SLURM&amp;#039;s &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; management, Mufasa makes use of Nvidia&amp;#039;s [https://developer.nvidia.com/nvidia-management-library-nvml NVML library]. For additional information see [https://slurm.schedmd.com/gres.html SLURM&amp;#039;s documentation].&lt;/div&gt;</summary>
		<author><name>GiulioFontana</name></author>
	</entry>
	<entry>
		<id>https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2601</id>
		<title>SLURM</title>
		<link rel="alternate" type="text/html" href="https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2601"/>
		<updated>2026-11-18T22:40:29Z</updated>

		<summary type="html">&lt;p&gt;GiulioFontana: /* Elements determining job priority */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page presents the features of SLURM that are most relevant to Mufasa&amp;#039;s [[Roles|Job Users]]. Job Users can submit jobs for execution, cancel their own jobs, and see other users&amp;#039; jobs (but not intervene on them).&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa &amp;#039;&amp;#039;&amp;#039;must use SLURM&amp;#039;&amp;#039;&amp;#039; to run resource-heavy processes, i.e. computing jobs that require one or more of the following:&lt;br /&gt;
* GPUs&lt;br /&gt;
* multiple CPUs&lt;br /&gt;
* powerful CPUs&lt;br /&gt;
* a significant amount of RAM&lt;br /&gt;
&lt;br /&gt;
In fact, only processes run via SLURM have access to all the resources of Mufasa. Processes run outside SLURM are executed by the [[System#Login server|login server]] virtual machine, which has minimal resources and no GPUs. Using SLURM is therefore the only way to execute resource-heavy jobs on Mufasa (this is a key difference between Mufasa 1.0 and Mufasa 2.0).&lt;br /&gt;
&lt;br /&gt;
= SLURM in a nutshell =&lt;br /&gt;
&lt;br /&gt;
Computation jobs on Mufasa needs to be launched via [[System#The SLURM job scheduling system|SLURM]]. SLURM provides jobs with access to the [[#System resources subjected to limitations|physical resources]] of Mufasa, such as CPUs, GPUs and RAM. Thanks to SLURM, processing jobs share system resources, optimising their occupation and availability. &lt;br /&gt;
&lt;br /&gt;
When a user runs a job, the job does not get executed immediately and is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039;. SLURM executes jobs according to their order in the queue: the top job in the queue gets executed as soon as the necessary resources are available, while jobs lower in the queue wait longer. The position of a job in the queue is due to the &amp;#039;&amp;#039;&amp;#039;[[#Job priority|priority]]&amp;#039;&amp;#039;&amp;#039; assigned to it by SLURM, with higher-priority jobs closer to the top. As a general rule:&lt;br /&gt;
&lt;br /&gt;
;: &amp;#039;&amp;#039;&amp;#039;the greater the fraction of Mufasa&amp;#039;s overall resources that a job asks for, the lower the job&amp;#039;s priority will be&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
The priority mechanism is used to encourage users to use Mufasa&amp;#039;s resources (i.e.: GPUs, CPUs, RAM, execution time) in an effective and equitable manner. This page includes a [[#How_to_maximise_the_priority_of_your_jobs|chart explaining how to maximise the priority of your jobs]].&lt;br /&gt;
&lt;br /&gt;
The &amp;#039;&amp;#039;&amp;#039;time&amp;#039;&amp;#039;&amp;#039; available to a job for its execution is controlled by SLURM. When a user requests execution of a job, they must specify the duration of the time slot that the job needs. The job must complete its execution before the end of the requested time slot, because at the end of such slot it gets killed by SLURM.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0 access to system resources is managed via SLURM&amp;#039;s [[#SLURM Quality of Service (QOS)|Quality of Service (QOS)]] mechanism (Mufasa 1.0 used [[#SLURM_partitions|partitions]] instead). While [[User Jobs#Running jobs with SLURM|launching a processing job via SLURM]], the user must always specify the chosen QOS. QOSes differ in the set of resources that they provide access to because each of them is designed to fit a given type of job.&lt;br /&gt;
&lt;br /&gt;
Mufasa sets limits to the number of jobs by the same user. This page includes a [[#Limits on jobs by the same user|table summarising such limits]].&lt;br /&gt;
&lt;br /&gt;
= SLURM Quality of Service (QOS) =&lt;br /&gt;
&lt;br /&gt;
Through &amp;#039;&amp;#039;&amp;#039;Quality of Services&amp;#039;&amp;#039;&amp;#039; (&amp;#039;&amp;#039;&amp;#039;QOSes&amp;#039;&amp;#039;&amp;#039;), SLURM lets system configurators assign a name to a set of related constraints.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, QOSes are used to define different levels of access to the server&amp;#039;s resources. When [[User Jobs|executing a job with SLURM]], a user must always &amp;#039;&amp;#039;&amp;#039;specify the QOS&amp;#039;&amp;#039;&amp;#039; that their job will use: this choice, in turn, determines what resources the job is able to access and influences the [[#Job priority|priority]] of the job.&lt;br /&gt;
&lt;br /&gt;
Mufasa&amp;#039;s QOSes and their features can be inspected with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list qos format=name%-11,priority,MaxSubmit,maxwall,maxtres%-80,mintres%-18&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following (see below for the same data in table form):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
Name          Priority MaxSubmit     MaxWall MaxTRES                                                                          MinTRES            &lt;br /&gt;
----------- ---------- --------- ----------- -------------------------------------------------------------------------------- ------------------ &lt;br /&gt;
normal               0                                                                                                                           &lt;br /&gt;
nogpu                4         1  3-00:00:00 cpu=16,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=128G                    &lt;br /&gt;
gpuheavy-20          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=2,mem=128G             gres/gpu:4g.20gb=1 &lt;br /&gt;
gpuheavy-40          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=1,gres/gpu:4g.20gb=0,mem=128G             gres/gpu:40gb=1    &lt;br /&gt;
gpulight             8         1    12:00:00 cpu=2,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpu                  2         1  1-00:00:00 cpu=8,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpuwide              2         2  1-00:00:00 cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=1,mem=64G              gres/gpu:4g.20gb=1 &lt;br /&gt;
build               32         1    02:00:00 cpu=2,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=16G&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The columns of this output are the following:&lt;br /&gt;
&lt;br /&gt;
:; Name&lt;br /&gt;
:: name of the QOS&lt;br /&gt;
&lt;br /&gt;
:; Priority&lt;br /&gt;
:: priority tier associated to the QOS (higher value = higher priority): see [[#Job priority|Job priority]] for details&lt;br /&gt;
&lt;br /&gt;
:; MaxSubmit&lt;br /&gt;
:: maximum number of jobs from a single user that can be submitted to SLURM with this QOS; submitted jobs include both running and queued jobs&lt;br /&gt;
:: See [[#Limits on jobs by the same user|Limits on jobs by the same user]] for an overview of the limits on jobs set by Mufasa.&lt;br /&gt;
&lt;br /&gt;
:; MaxWall&lt;br /&gt;
:: maximum wall clock duration of the jobs using the QOS (after which they are killed by SLURM), in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
:: For some QOSes these are not set: it means that they are determined by the [[#SLURM partitions|partition]]. Partitions also define the [[#Default values|default duration]] of jobs.&lt;br /&gt;
&lt;br /&gt;
:; MaxTRES&lt;br /&gt;
:: amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) available to a job using the QOS, where&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;cpu=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of CPUs (i.e., processor cores) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of CPUs specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;gres/&amp;#039;&amp;#039;gpu:Type&amp;#039;&amp;#039;=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of GPUs of class &amp;lt;code&amp;gt;&amp;#039;&amp;#039;Type&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; (see [[User Jobs#gres syntax|&amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax]]) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; (for QOSes that allow access to GPUs) if not specified, the job cannot be launched&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;mem=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;G&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum amount of system RAM is &amp;#039;&amp;#039;K&amp;#039;&amp;#039; GBytes&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of RAM specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
&lt;br /&gt;
:; MinTRES&lt;br /&gt;
:: minimum amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) that a job using the QOS must request in order to actually get executed by SLURM.&lt;br /&gt;
:: (If your job does not actually need these resources, you&amp;#039;ve chosen the wrong QOS: you can use one with a higher priority.)&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; QOS is the default one, and exists only to ensure that users always specify a QOS when running a job. Since &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; has zero priority and no resources, a job run using this QOS would never be run.&lt;br /&gt;
&lt;br /&gt;
The information provided by the &amp;lt;code&amp;gt;sacctmgr list qos&amp;lt;/code&amp;gt; command above is summarised by the following table:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! QOS&lt;br /&gt;
! Priority&amp;lt;/br&amp;gt;tier&lt;br /&gt;
! Max&amp;lt;/br&amp;gt;Submit&lt;br /&gt;
! MaxWall&amp;lt;/br&amp;gt;[h]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;CPUs&lt;br /&gt;
! max RAM&amp;lt;/br&amp;gt;[GB]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;3g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;4g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;40GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | build&lt;br /&gt;
| 32&lt;br /&gt;
| 1&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 16&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpulight&lt;br /&gt;
| 8&lt;br /&gt;
| 1&lt;br /&gt;
| 12&lt;br /&gt;
| 2&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | nogpu&lt;br /&gt;
| 4&lt;br /&gt;
| 1&lt;br /&gt;
| 72&lt;br /&gt;
| 16&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpu&lt;br /&gt;
| 2&lt;br /&gt;
| 1&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuwide&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-20&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| 2 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-40&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
A key piece of information in the table above is the &amp;#039;&amp;#039;&amp;#039;priority tier&amp;#039;&amp;#039;&amp;#039; associated to each QOS. In Mufasa 2.0, priority tiers are used to encourage users to use the &amp;#039;&amp;#039;&amp;#039;least powerful QOS that is compatible with their needs&amp;#039;&amp;#039;&amp;#039;, where &amp;quot;powerful&amp;quot; means &amp;quot;rich with resources&amp;quot;. Less powerful QOSes increase the priority of the jobs that use them, so these jobs tend to be executed sooner. See [[#Job priority|Job priority]] for details about how priority affects the execution order of jobs in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
== The &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS ==&lt;br /&gt;
&lt;br /&gt;
This QOS is specifically designed to be used by Mufasa users to &amp;#039;&amp;#039;&amp;#039;quickly build [[System#Containers|container images]]&amp;#039;&amp;#039;&amp;#039;. Its associated priority tier is very high: this, combined with the fact that jobs using this QOS require few resources, means that such jobs usually get executed very soon.&lt;br /&gt;
&lt;br /&gt;
On the other hand, the limited resources and the lack of access to the GPUs make the &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS unsuitable for other tasks.&lt;br /&gt;
&lt;br /&gt;
See [[Singularity#Building Singularity images|Building Singularity images]] for directions about building Singularity container images.&lt;br /&gt;
&lt;br /&gt;
== Restricted QOSes ==&lt;br /&gt;
&lt;br /&gt;
In Mufasa, the most powerful QOSes are reserved to researchers (called &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users: these include academic personnel and Ph.D. students). M.Sc. students, called &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users, cannot use them while running jobs.&lt;br /&gt;
&lt;br /&gt;
See [[#research users and students users|below]] to understand how &amp;#039;&amp;#039;user categories&amp;#039;&amp;#039; work in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
= &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users =&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa belong to two &amp;#039;&amp;#039;&amp;#039;user categories&amp;#039;&amp;#039;&amp;#039;. Categories provide users with different access to  to Mufasa&amp;#039;s resources: the idea being to provide researchers with more access while still letting students benefit from the server&amp;#039;s capabilities.&lt;br /&gt;
&lt;br /&gt;
User categories in Mufasa are the following:&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. academic personnel and Ph.D. students&lt;br /&gt;
::: * can use all QOSes, including the the [[#Restricted QOSes|restricted ones]]&lt;br /&gt;
::: * their jobs have a higher &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a higher number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. M.Sc. students&lt;br /&gt;
::: * cannot use the [[#Restricted QOSes|restricted QOSes]]&lt;br /&gt;
::: * their jobs have a lower &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a lower number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
You can inspect the differences in priority and running jobs between &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association format=&amp;quot;account,priority,maxjobs&amp;quot; | grep -E &amp;#039;Account|research|students&amp;#039;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
   Account   Priority MaxJobs &lt;br /&gt;
  research          4       2 &lt;br /&gt;
  students          1       1&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
To know what limits apply to your own user, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association where user=$USER format=&amp;quot;user,priority,maxjobs,qos%-60&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
      User   Priority MaxJobs QOS                                                          &lt;br /&gt;
---------- ---------- ------- ------------------------------------------------------------ &lt;br /&gt;
    preali          4       2 build,gpu,gpuheavy-20,gpuheavy-40,gpulight,gpuwide,nogpu&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The list under &amp;quot;QOS&amp;quot; shows what QOSes your user is allowed to use when [[User Jobs|running jobs]]. &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users can use all of them, while &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users can only access a subset of them.&lt;br /&gt;
&lt;br /&gt;
= Limits on jobs by the same user =&lt;br /&gt;
&lt;br /&gt;
Mufasa enforces limits on the number of jobs from a single user. Such limits aim at preventing users from &amp;quot;hogging&amp;quot; system resources, and apply to:&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;submitted jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that the user asked SLURM to execute, each of which may currently be either running or queued&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;running jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that are currently in execution&lt;br /&gt;
&lt;br /&gt;
Limits on submitted jobs are set via [[#SLURM Quality of Service (QOS)|QOSes]], while limits on running jobs are set via [[#research users and students users|user category]].&lt;br /&gt;
&lt;br /&gt;
The following table summarises what limits exist:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! &lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;running&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;submitted&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | global limits&amp;lt;br/&amp;gt;(system-wide)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to limits set by individual QOSes (below)&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | limits for&amp;lt;br/&amp;gt;individual&amp;lt;/br&amp;gt;QOSes&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to global limits (above)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for the&amp;#039;&amp;#039; &amp;lt;code&amp;gt;gpuwide&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;QOS&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for each of the other QOSes&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
= Job priority =&lt;br /&gt;
&lt;br /&gt;
Once the execution of a job has been requested, the job is not run immediately: it is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039; by SLURM, together with all the other jobs awaiting execution. The job on top of the queue at any time is the first to be put into execution as soon as the resources it requires are available. &lt;br /&gt;
&lt;br /&gt;
The order of the items in the job queue depends on their &amp;#039;&amp;#039;&amp;#039;priority&amp;#039;&amp;#039;&amp;#039;. The job with the highest priority is put by SLURM on top of the queue, while all other jobs are put in the queue in order of descending priority.&lt;br /&gt;
&lt;br /&gt;
The goal of SLURM is to maximise resource availability: i.e., to ensure the shorter possible wait time before job execution. To achieve this goal, in Mufasa SLURM is configured to &amp;#039;&amp;#039;&amp;#039;encourage users not to ask for resources or execution time that their job doesn&amp;#039;t need&amp;#039;&amp;#039;&amp;#039;. This is done via the priority mechanism: the more resources and/or time a job requests, the lower its priority will be; and the later it will be executed.&lt;br /&gt;
&lt;br /&gt;
Priority management in Mufasa is designed to set up a &amp;#039;&amp;#039;&amp;#039;virtuous cycle&amp;#039;&amp;#039;&amp;#039; where users, by carefully choosing what to ask for, obtain two results:&lt;br /&gt;
* they ensure that their job is executed as soon as possible;&lt;br /&gt;
* they leave as much as possible of Mufasa&amp;#039;s resources free for other users&amp;#039;s jobs.&lt;br /&gt;
&lt;br /&gt;
== Elements determining job priority ==&lt;br /&gt;
In Mufasa, the priority of a job is computed by SLURM according to the following elements:&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#research users and students users|User category]]&amp;#039;&amp;#039;&amp;#039; (i.e., &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;)&lt;br /&gt;
::: Used to provide higher priority to jobs run by &amp;#039;&amp;#039;&amp;#039;research personnel&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#SLURM Quality of Service (QOS)|QOS]]&amp;#039;&amp;#039;&amp;#039; used by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;less resources&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Number of CPUs&amp;#039;&amp;#039;&amp;#039; requested by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;a lower number of CPUs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job Age&amp;#039;&amp;#039;&amp;#039;, i.e. the time that the job has been waiting in the queue&lt;br /&gt;
::: Used to provide higher priority to jobs which have been &amp;#039;&amp;#039;&amp;#039;queued for a longer time&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
::: (Periods when a queued job is not eligible for running because of the [[#Limits on jobs by the same user|limits on running jobs by the same user]] do not increase job age)&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;FairShare&amp;#039;&amp;#039;&amp;#039;, i.e. a factor computed by SLURM to balance use of the system by different users&lt;br /&gt;
::: Used to provide higher priority to jobs by users who &amp;#039;&amp;#039;&amp;#039;used Mufasa&amp;#039;s resources less than others&amp;#039;&amp;#039;&amp;#039; (see [[#How FairShare works|below]])&lt;br /&gt;
&lt;br /&gt;
You can understand how each of these elements contributes to the priority of the jobs waiting for execution (both yours and of other users) with command &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;. A tailored version of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; that only displays the parameters listed above is the following:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sprio -o &amp;quot;%6i %9u %.8Y %.10B %.5Q %.9T %.5A %.5F&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Here is an example of the output of the command above:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
JOBID  USER      PRIORITY      ASSOC   QOS      TRES   AGE FAIRS&lt;br /&gt;
5790   amoglia       3495       2500   375  cpu=-800   933   487&lt;br /&gt;
5791   qliu          3010       2500   375  cpu=-800   916    19&lt;br /&gt;
5817   mzoccola      2336        625   750  cpu=-800   495  1266&lt;br /&gt;
5822   fbottazzi     2235        625   750  cpu=-600   467   994&lt;br /&gt;
5837   lpassoni      2508       2500   750  cpu=-800     0    58&lt;br /&gt;
5838   bagostini     4437        625  3000  cpu=-200     0  1013&lt;br /&gt;
5839   flazzaro      1549        625   750  cpu=-800     0   974&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
In it, column PRIORITY provides the overall priority of each job. Each priority value is the sum of the five values to its right; in particular, column ASSOC corresponds to user category, while column TRES corresponds to the number of CPUs. Note that the contribution of TRES to priority is &amp;#039;&amp;#039;negative&amp;#039;&amp;#039;, i.e. it is a penalty; its value is proportional to the number of CPUs requested by the job.&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;Things to consider when using &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
* Its output is only relevant for jobs &amp;#039;&amp;#039;not yet in execution&amp;#039;&amp;#039;. For an already running job, the values shown by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; correspond to the priority that the job would get if evaluated now: however, this priority does not affect job execution in any way.&lt;br /&gt;
* Jobs listed by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; may temporarily differ from jobs listed by &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;: this is due to the fact that SLURM updates its internal data (used by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;) every few minutes, while the output of &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt; is updated much more frequently.&lt;br /&gt;
&lt;br /&gt;
=== How FairShare works ===&lt;br /&gt;
In Mufasa 2.0, the FairShare of each user is computed according to these rules:&lt;br /&gt;
# It considers the quantity used by (current and past) user jobs of:&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;CPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of CPUs used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;RAM&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the amount of RAM used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;GPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of GPUs used&lt;br /&gt;
#** note: 40 GB GPUs have double the impact on FairShare than 20 GB GPUs&lt;br /&gt;
# The impact on FairShare of any use of resources is &amp;#039;&amp;#039;&amp;#039;proportional to its duration&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
#* example: using 32 GB of RAM for 48h has the same impact of using 64 GB of RAM for 24h&lt;br /&gt;
# FairShare has a &amp;quot;fading memory&amp;quot;: i.e., resource use has less and less impact on FairShare the farther in the past it is&lt;br /&gt;
#* consequence: over time, the &amp;quot;history&amp;quot; of a user gets forgotten by FairShare&lt;br /&gt;
&lt;br /&gt;
== How to maximise the priority of your jobs ==&lt;br /&gt;
&lt;br /&gt;
Every time you run a SLURM job, follow these guidelines:&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Choose the less powerful QOS compatible with the needs of your job&lt;br /&gt;
:: QOSes with access to less resources have higher base priority.&lt;br /&gt;
&lt;br /&gt;
; Only request CPUs and RAM that your job really needs&lt;br /&gt;
:: Usually, code exploits multiple CPUs only if specifically designed to do so: if unsure, only ask for 1 CPU.&lt;br /&gt;
:: The fewer resources your job asks for, the less it will wait before they become available.&lt;br /&gt;
:: The more CPUs your job asks for, the stronger the penalty to its priority.&lt;br /&gt;
:: Asking for fewer CPUs and/or less RAM improves your FairShare.&lt;br /&gt;
&lt;br /&gt;
; Do not request more time than your jobs needs to complete&lt;br /&gt;
:: Make a worst-case estimate and only ask for that duration.&lt;br /&gt;
:: Asking for less of Mufasa&amp;#039;s time improves your FairShare.&lt;br /&gt;
&lt;br /&gt;
; Debug and test code using less powerful QOSes before running it with more powerful QOSes&lt;br /&gt;
:: Your test jobs will get a higher priority and your FairShare will improve.&lt;br /&gt;
&lt;br /&gt;
; Cancel jobs when you don&amp;#039;t need them anymore&lt;br /&gt;
:: Always use [[User_Jobs#Cancelling_a_job_with_scancel|scancel]] to delete completed (or crashed) jobs: your Fairshare will improve.&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Protip: it&amp;#039;s a good idea to [[User Jobs#Looking for unused GPUs|check for unused GPUs]] before choosing what to request. Requesting a GPU that is currently idle will help your job get executed sooner.&lt;br /&gt;
&lt;br /&gt;
= SLURM partitions =&lt;br /&gt;
&lt;br /&gt;
Partitions are another mechanism provided by SLURM to create different levels of access to system resources. Since in Mufasa 2.0 access to resources is controlled via [[#SLURM Quality of Service (QOS)|QOSes]], partitions are not very relevant. (This is one of the main differences between Mufasa 1.0 and Mufasa 2.0.)&lt;br /&gt;
&lt;br /&gt;
Note, however, that the default values for some features of SLURM jobs (e.g., duration) are [[#Default values|set by the partition]].&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, there is a single SLURM partition, called &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt;, and all jobs run on it. The state of &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt; can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sinfo -o &amp;quot;%10P %5a %9T %11L %10l&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PARTITION  AVAIL STATE     DEFAULTTIME TIMELIMIT &lt;br /&gt;
jobs*      up    idle      1:00:00     3-00:00:00&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where columns correspond to the following information:&lt;br /&gt;
&lt;br /&gt;
:; PARTITION&lt;br /&gt;
:: name of the partition; the asterisks indicates that it&amp;#039;s the default one&lt;br /&gt;
&lt;br /&gt;
:; AVAIL&lt;br /&gt;
:: state/availability of the partition: see [[#Partition availability|below]]&lt;br /&gt;
&lt;br /&gt;
:; STATE&lt;br /&gt;
:: state (using [https://slurm.schedmd.com/sinfo.html#SECTION_NODE-STATE-CODES these codes])&lt;br /&gt;
:: typical values are &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;mixed&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that some of the resources are busy executing jobs while other are idle, and &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;allocated&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that all of the resources are in use&lt;br /&gt;
&lt;br /&gt;
:; DEFAULTTIME&lt;br /&gt;
:: default runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:; TIMELIMIT&lt;br /&gt;
:: maximum runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
The asterisk at the end of the partition name indicates the default partition, i.e. the one on which jobs which do not ask for a specific partition are run.&lt;br /&gt;
&lt;br /&gt;
Command &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; does not tell you about the &amp;#039;&amp;#039;jobs&amp;#039;&amp;#039; submitted to a partition. This information is obtained, instead, with [[User Jobs#Inspecting jobs with squeue|command &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;]].&lt;br /&gt;
&lt;br /&gt;
== Partition availability ==&lt;br /&gt;
&lt;br /&gt;
The most important information that &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; provides is the &amp;#039;&amp;#039;&amp;#039;availability&amp;#039;&amp;#039;&amp;#039; (also called &amp;#039;&amp;#039;&amp;#039;state&amp;#039;&amp;#039;&amp;#039;) of partitions. This is shown in column &amp;quot;AVAIL&amp;quot;. Possible partition states are:&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is available&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed as soon as resources allow&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is in the process of becoming unavailable (i.e., of entering the &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; state: see below)&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is unavailable&lt;br /&gt;
:: There are no running jobs&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
When a partition goes from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; no harm is done to running jobs. In a normally functioning SLURM system, the passage from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; happens only when no jobs are running on the partition. If (e.g., due to a malfunction) the passage happens with jobs still running, they get killed.&lt;br /&gt;
&lt;br /&gt;
A partition in state &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; requires intervention by a [[Roles|Job Administrator]] to be restored to &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
== Default values ==&lt;br /&gt;
&lt;br /&gt;
The features of SLURM partitions, including the &amp;#039;&amp;#039;&amp;#039;default values&amp;#039;&amp;#039;&amp;#039; which are applied to jobs that do not make explicit requests, can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scontrol show partition&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PartitionName=jobs&lt;br /&gt;
   AllowGroups=ALL AllowAccounts=ALL AllowQos=nogpu,gpulight,gpu,gpuwide,gpuheavy-20,gpuheavy-40&lt;br /&gt;
   AllocNodes=ALL Default=YES QoS=N/A&lt;br /&gt;
   DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO&lt;br /&gt;
   MaxNodes=UNLIMITED MaxTime=3-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED&lt;br /&gt;
   Nodes=gn01&lt;br /&gt;
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO&lt;br /&gt;
   OverTimeLimit=NONE PreemptMode=OFF&lt;br /&gt;
   State=UP TotalCPUs=48 TotalNodes=1 SelectTypeParameters=NONE&lt;br /&gt;
   JobDefaults=(null)&lt;br /&gt;
   DefMemPerNode=4096 MaxMemPerNode=UNLIMITED&lt;br /&gt;
   TRES=cpu=48,mem=1011435M,node=1,billing=49,gres/gpu=13,gres/gpu:3g.20gb=5,gres/gpu:40gb=3,gres/gpu:4g.20gb=5&lt;br /&gt;
   TRESBillingWeights=cpu=1.0,gres/gpu:3g.20gb=6.0,gres/gpu:4g.20gb=6.0,gres/gpu:40gb=6.0,mem=0.05g&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In the output of &amp;lt;code&amp;gt;scontrol show partition&amp;lt;/code&amp;gt; the most relevant default values for Mufasa users are:&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefaultTime&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default execution time assigned to a job run on the partition (e.g., 1 hour)&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefMemPerNode&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default amount of RAM assigned to a job run on the partition (e.g., 4GB)&lt;br /&gt;
&lt;br /&gt;
= System resources subjected to limitations =&lt;br /&gt;
&lt;br /&gt;
In systems based on SLURM like Mufasa, &amp;#039;&amp;#039;&amp;#039;TRES (Trackable RESources)&amp;#039;&amp;#039;&amp;#039; are (from [https://slurm.schedmd.com/tres.html SLURM&amp;#039;s documentation] &amp;quot;&amp;#039;&amp;#039;resources that can be tracked for usage or used to enforce limits against.&amp;#039;&amp;#039;&amp;quot;&lt;br /&gt;
&lt;br /&gt;
TRES include CPUs, RAM and &amp;#039;&amp;#039;&amp;#039;GRES&amp;#039;&amp;#039;&amp;#039;. The last term stands for &amp;#039;&amp;#039;Generic RESources&amp;#039;&amp;#039; that a job may need for its execution. In Mufasa, the only &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; resources are the GPUs.&lt;br /&gt;
&lt;br /&gt;
== &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax ==&lt;br /&gt;
&lt;br /&gt;
To ask SLURM to assign GRES resources (i.e., GPUs) to a job, a special syntax must be used. Precisely, the name of each GPU resource takes the form&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:Name:Type&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
Considering the [[System#CPUs and GPUs|GPU complement of Mufasa]], &amp;lt;code&amp;gt;Type&amp;lt;/code&amp;gt; takes the following values:&lt;br /&gt;
&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:40gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 40 Gbytes of RAM&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:4g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 4 compute units&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 3 compute units&lt;br /&gt;
&lt;br /&gt;
So, for instance,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
identifies a resource corresponding to a GPU with 20 GB of RAM and 3 compute units.&lt;br /&gt;
&lt;br /&gt;
When asking for a GRES resource (e.g., in an &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command or an &amp;lt;code&amp;gt;SBATCH&amp;lt;/code&amp;gt; directive of an [[User Jobs#Using execution scripts to run jobs|execution script]]), the syntax required by SLURM is&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:&amp;lt;Type&amp;gt;:&amp;lt;Quantity&amp;gt;&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
where &amp;lt;code&amp;gt;Quantity&amp;lt;/code&amp;gt; is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of type &amp;lt;code&amp;gt;4g.20gb&amp;lt;/code&amp;gt; the syntax is&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:4g.20gb:2&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SLURM&amp;#039;s &amp;#039;&amp;#039;generic resources&amp;#039;&amp;#039; are defined in &amp;lt;code&amp;gt;/etc/slurm/gres.conf&amp;lt;/code&amp;gt;. In order to make GPUs available to SLURM&amp;#039;s &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; management, Mufasa makes use of Nvidia&amp;#039;s [https://developer.nvidia.com/nvidia-management-library-nvml NVML library]. For additional information see [https://slurm.schedmd.com/gres.html SLURM&amp;#039;s documentation].&lt;/div&gt;</summary>
		<author><name>GiulioFontana</name></author>
	</entry>
	<entry>
		<id>https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2600</id>
		<title>SLURM</title>
		<link rel="alternate" type="text/html" href="https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2600"/>
		<updated>2026-11-18T22:37:56Z</updated>

		<summary type="html">&lt;p&gt;GiulioFontana: /* How to maximise the priority of your jobs */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page presents the features of SLURM that are most relevant to Mufasa&amp;#039;s [[Roles|Job Users]]. Job Users can submit jobs for execution, cancel their own jobs, and see other users&amp;#039; jobs (but not intervene on them).&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa &amp;#039;&amp;#039;&amp;#039;must use SLURM&amp;#039;&amp;#039;&amp;#039; to run resource-heavy processes, i.e. computing jobs that require one or more of the following:&lt;br /&gt;
* GPUs&lt;br /&gt;
* multiple CPUs&lt;br /&gt;
* powerful CPUs&lt;br /&gt;
* a significant amount of RAM&lt;br /&gt;
&lt;br /&gt;
In fact, only processes run via SLURM have access to all the resources of Mufasa. Processes run outside SLURM are executed by the [[System#Login server|login server]] virtual machine, which has minimal resources and no GPUs. Using SLURM is therefore the only way to execute resource-heavy jobs on Mufasa (this is a key difference between Mufasa 1.0 and Mufasa 2.0).&lt;br /&gt;
&lt;br /&gt;
= SLURM in a nutshell =&lt;br /&gt;
&lt;br /&gt;
Computation jobs on Mufasa needs to be launched via [[System#The SLURM job scheduling system|SLURM]]. SLURM provides jobs with access to the [[#System resources subjected to limitations|physical resources]] of Mufasa, such as CPUs, GPUs and RAM. Thanks to SLURM, processing jobs share system resources, optimising their occupation and availability. &lt;br /&gt;
&lt;br /&gt;
When a user runs a job, the job does not get executed immediately and is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039;. SLURM executes jobs according to their order in the queue: the top job in the queue gets executed as soon as the necessary resources are available, while jobs lower in the queue wait longer. The position of a job in the queue is due to the &amp;#039;&amp;#039;&amp;#039;[[#Job priority|priority]]&amp;#039;&amp;#039;&amp;#039; assigned to it by SLURM, with higher-priority jobs closer to the top. As a general rule:&lt;br /&gt;
&lt;br /&gt;
;: &amp;#039;&amp;#039;&amp;#039;the greater the fraction of Mufasa&amp;#039;s overall resources that a job asks for, the lower the job&amp;#039;s priority will be&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
The priority mechanism is used to encourage users to use Mufasa&amp;#039;s resources (i.e.: GPUs, CPUs, RAM, execution time) in an effective and equitable manner. This page includes a [[#How_to_maximise_the_priority_of_your_jobs|chart explaining how to maximise the priority of your jobs]].&lt;br /&gt;
&lt;br /&gt;
The &amp;#039;&amp;#039;&amp;#039;time&amp;#039;&amp;#039;&amp;#039; available to a job for its execution is controlled by SLURM. When a user requests execution of a job, they must specify the duration of the time slot that the job needs. The job must complete its execution before the end of the requested time slot, because at the end of such slot it gets killed by SLURM.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0 access to system resources is managed via SLURM&amp;#039;s [[#SLURM Quality of Service (QOS)|Quality of Service (QOS)]] mechanism (Mufasa 1.0 used [[#SLURM_partitions|partitions]] instead). While [[User Jobs#Running jobs with SLURM|launching a processing job via SLURM]], the user must always specify the chosen QOS. QOSes differ in the set of resources that they provide access to because each of them is designed to fit a given type of job.&lt;br /&gt;
&lt;br /&gt;
Mufasa sets limits to the number of jobs by the same user. This page includes a [[#Limits on jobs by the same user|table summarising such limits]].&lt;br /&gt;
&lt;br /&gt;
= SLURM Quality of Service (QOS) =&lt;br /&gt;
&lt;br /&gt;
Through &amp;#039;&amp;#039;&amp;#039;Quality of Services&amp;#039;&amp;#039;&amp;#039; (&amp;#039;&amp;#039;&amp;#039;QOSes&amp;#039;&amp;#039;&amp;#039;), SLURM lets system configurators assign a name to a set of related constraints.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, QOSes are used to define different levels of access to the server&amp;#039;s resources. When [[User Jobs|executing a job with SLURM]], a user must always &amp;#039;&amp;#039;&amp;#039;specify the QOS&amp;#039;&amp;#039;&amp;#039; that their job will use: this choice, in turn, determines what resources the job is able to access and influences the [[#Job priority|priority]] of the job.&lt;br /&gt;
&lt;br /&gt;
Mufasa&amp;#039;s QOSes and their features can be inspected with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list qos format=name%-11,priority,MaxSubmit,maxwall,maxtres%-80,mintres%-18&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following (see below for the same data in table form):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
Name          Priority MaxSubmit     MaxWall MaxTRES                                                                          MinTRES            &lt;br /&gt;
----------- ---------- --------- ----------- -------------------------------------------------------------------------------- ------------------ &lt;br /&gt;
normal               0                                                                                                                           &lt;br /&gt;
nogpu                4         1  3-00:00:00 cpu=16,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=128G                    &lt;br /&gt;
gpuheavy-20          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=2,mem=128G             gres/gpu:4g.20gb=1 &lt;br /&gt;
gpuheavy-40          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=1,gres/gpu:4g.20gb=0,mem=128G             gres/gpu:40gb=1    &lt;br /&gt;
gpulight             8         1    12:00:00 cpu=2,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpu                  2         1  1-00:00:00 cpu=8,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpuwide              2         2  1-00:00:00 cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=1,mem=64G              gres/gpu:4g.20gb=1 &lt;br /&gt;
build               32         1    02:00:00 cpu=2,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=16G&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The columns of this output are the following:&lt;br /&gt;
&lt;br /&gt;
:; Name&lt;br /&gt;
:: name of the QOS&lt;br /&gt;
&lt;br /&gt;
:; Priority&lt;br /&gt;
:: priority tier associated to the QOS (higher value = higher priority): see [[#Job priority|Job priority]] for details&lt;br /&gt;
&lt;br /&gt;
:; MaxSubmit&lt;br /&gt;
:: maximum number of jobs from a single user that can be submitted to SLURM with this QOS; submitted jobs include both running and queued jobs&lt;br /&gt;
:: See [[#Limits on jobs by the same user|Limits on jobs by the same user]] for an overview of the limits on jobs set by Mufasa.&lt;br /&gt;
&lt;br /&gt;
:; MaxWall&lt;br /&gt;
:: maximum wall clock duration of the jobs using the QOS (after which they are killed by SLURM), in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
:: For some QOSes these are not set: it means that they are determined by the [[#SLURM partitions|partition]]. Partitions also define the [[#Default values|default duration]] of jobs.&lt;br /&gt;
&lt;br /&gt;
:; MaxTRES&lt;br /&gt;
:: amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) available to a job using the QOS, where&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;cpu=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of CPUs (i.e., processor cores) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of CPUs specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;gres/&amp;#039;&amp;#039;gpu:Type&amp;#039;&amp;#039;=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of GPUs of class &amp;lt;code&amp;gt;&amp;#039;&amp;#039;Type&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; (see [[User Jobs#gres syntax|&amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax]]) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; (for QOSes that allow access to GPUs) if not specified, the job cannot be launched&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;mem=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;G&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum amount of system RAM is &amp;#039;&amp;#039;K&amp;#039;&amp;#039; GBytes&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of RAM specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
&lt;br /&gt;
:; MinTRES&lt;br /&gt;
:: minimum amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) that a job using the QOS must request in order to actually get executed by SLURM.&lt;br /&gt;
:: (If your job does not actually need these resources, you&amp;#039;ve chosen the wrong QOS: you can use one with a higher priority.)&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; QOS is the default one, and exists only to ensure that users always specify a QOS when running a job. Since &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; has zero priority and no resources, a job run using this QOS would never be run.&lt;br /&gt;
&lt;br /&gt;
The information provided by the &amp;lt;code&amp;gt;sacctmgr list qos&amp;lt;/code&amp;gt; command above is summarised by the following table:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! QOS&lt;br /&gt;
! Priority&amp;lt;/br&amp;gt;tier&lt;br /&gt;
! Max&amp;lt;/br&amp;gt;Submit&lt;br /&gt;
! MaxWall&amp;lt;/br&amp;gt;[h]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;CPUs&lt;br /&gt;
! max RAM&amp;lt;/br&amp;gt;[GB]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;3g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;4g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;40GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | build&lt;br /&gt;
| 32&lt;br /&gt;
| 1&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 16&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpulight&lt;br /&gt;
| 8&lt;br /&gt;
| 1&lt;br /&gt;
| 12&lt;br /&gt;
| 2&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | nogpu&lt;br /&gt;
| 4&lt;br /&gt;
| 1&lt;br /&gt;
| 72&lt;br /&gt;
| 16&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpu&lt;br /&gt;
| 2&lt;br /&gt;
| 1&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuwide&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-20&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| 2 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-40&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
A key piece of information in the table above is the &amp;#039;&amp;#039;&amp;#039;priority tier&amp;#039;&amp;#039;&amp;#039; associated to each QOS. In Mufasa 2.0, priority tiers are used to encourage users to use the &amp;#039;&amp;#039;&amp;#039;least powerful QOS that is compatible with their needs&amp;#039;&amp;#039;&amp;#039;, where &amp;quot;powerful&amp;quot; means &amp;quot;rich with resources&amp;quot;. Less powerful QOSes increase the priority of the jobs that use them, so these jobs tend to be executed sooner. See [[#Job priority|Job priority]] for details about how priority affects the execution order of jobs in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
== The &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS ==&lt;br /&gt;
&lt;br /&gt;
This QOS is specifically designed to be used by Mufasa users to &amp;#039;&amp;#039;&amp;#039;quickly build [[System#Containers|container images]]&amp;#039;&amp;#039;&amp;#039;. Its associated priority tier is very high: this, combined with the fact that jobs using this QOS require few resources, means that such jobs usually get executed very soon.&lt;br /&gt;
&lt;br /&gt;
On the other hand, the limited resources and the lack of access to the GPUs make the &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS unsuitable for other tasks.&lt;br /&gt;
&lt;br /&gt;
See [[Singularity#Building Singularity images|Building Singularity images]] for directions about building Singularity container images.&lt;br /&gt;
&lt;br /&gt;
== Restricted QOSes ==&lt;br /&gt;
&lt;br /&gt;
In Mufasa, the most powerful QOSes are reserved to researchers (called &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users: these include academic personnel and Ph.D. students). M.Sc. students, called &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users, cannot use them while running jobs.&lt;br /&gt;
&lt;br /&gt;
See [[#research users and students users|below]] to understand how &amp;#039;&amp;#039;user categories&amp;#039;&amp;#039; work in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
= &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users =&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa belong to two &amp;#039;&amp;#039;&amp;#039;user categories&amp;#039;&amp;#039;&amp;#039;. Categories provide users with different access to  to Mufasa&amp;#039;s resources: the idea being to provide researchers with more access while still letting students benefit from the server&amp;#039;s capabilities.&lt;br /&gt;
&lt;br /&gt;
User categories in Mufasa are the following:&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. academic personnel and Ph.D. students&lt;br /&gt;
::: * can use all QOSes, including the the [[#Restricted QOSes|restricted ones]]&lt;br /&gt;
::: * their jobs have a higher &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a higher number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. M.Sc. students&lt;br /&gt;
::: * cannot use the [[#Restricted QOSes|restricted QOSes]]&lt;br /&gt;
::: * their jobs have a lower &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a lower number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
You can inspect the differences in priority and running jobs between &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association format=&amp;quot;account,priority,maxjobs&amp;quot; | grep -E &amp;#039;Account|research|students&amp;#039;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
   Account   Priority MaxJobs &lt;br /&gt;
  research          4       2 &lt;br /&gt;
  students          1       1&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
To know what limits apply to your own user, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association where user=$USER format=&amp;quot;user,priority,maxjobs,qos%-60&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
      User   Priority MaxJobs QOS                                                          &lt;br /&gt;
---------- ---------- ------- ------------------------------------------------------------ &lt;br /&gt;
    preali          4       2 build,gpu,gpuheavy-20,gpuheavy-40,gpulight,gpuwide,nogpu&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The list under &amp;quot;QOS&amp;quot; shows what QOSes your user is allowed to use when [[User Jobs|running jobs]]. &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users can use all of them, while &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users can only access a subset of them.&lt;br /&gt;
&lt;br /&gt;
= Limits on jobs by the same user =&lt;br /&gt;
&lt;br /&gt;
Mufasa enforces limits on the number of jobs from a single user. Such limits aim at preventing users from &amp;quot;hogging&amp;quot; system resources, and apply to:&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;submitted jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that the user asked SLURM to execute, each of which may currently be either running or queued&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;running jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that are currently in execution&lt;br /&gt;
&lt;br /&gt;
Limits on submitted jobs are set via [[#SLURM Quality of Service (QOS)|QOSes]], while limits on running jobs are set via [[#research users and students users|user category]].&lt;br /&gt;
&lt;br /&gt;
The following table summarises what limits exist:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! &lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;running&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;submitted&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | global limits&amp;lt;br/&amp;gt;(system-wide)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to limits set by individual QOSes (below)&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | limits for&amp;lt;br/&amp;gt;individual&amp;lt;/br&amp;gt;QOSes&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to global limits (above)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for the&amp;#039;&amp;#039; &amp;lt;code&amp;gt;gpuwide&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;QOS&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for each of the other QOSes&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
= Job priority =&lt;br /&gt;
&lt;br /&gt;
Once the execution of a job has been requested, the job is not run immediately: it is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039; by SLURM, together with all the other jobs awaiting execution. The job on top of the queue at any time is the first to be put into execution as soon as the resources it requires are available. &lt;br /&gt;
&lt;br /&gt;
The order of the items in the job queue depends on their &amp;#039;&amp;#039;&amp;#039;priority&amp;#039;&amp;#039;&amp;#039;. The job with the highest priority is put by SLURM on top of the queue, while all other jobs are put in the queue in order of descending priority.&lt;br /&gt;
&lt;br /&gt;
The goal of SLURM is to maximise resource availability: i.e., to ensure the shorter possible wait time before job execution. To achieve this goal, in Mufasa SLURM is configured to &amp;#039;&amp;#039;&amp;#039;encourage users not to ask for resources or execution time that their job doesn&amp;#039;t need&amp;#039;&amp;#039;&amp;#039;. This is done via the priority mechanism: the more resources and/or time a job requests, the lower its priority will be; and the later it will be executed.&lt;br /&gt;
&lt;br /&gt;
Priority management in Mufasa is designed to set up a &amp;#039;&amp;#039;&amp;#039;virtuous cycle&amp;#039;&amp;#039;&amp;#039; where users, by carefully choosing what to ask for, obtain two results:&lt;br /&gt;
* they ensure that their job is executed as soon as possible;&lt;br /&gt;
* they leave as much as possible of Mufasa&amp;#039;s resources free for other users&amp;#039;s jobs.&lt;br /&gt;
&lt;br /&gt;
== Elements determining job priority ==&lt;br /&gt;
In Mufasa, the priority of a job is computed by SLURM according to the following elements:&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#research users and students users|User category]]&amp;#039;&amp;#039;&amp;#039; (i.e., &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;)&lt;br /&gt;
::: Used to provide higher priority to jobs run by &amp;#039;&amp;#039;&amp;#039;research personnel&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#SLURM Quality of Service (QOS)|QOS]]&amp;#039;&amp;#039;&amp;#039; used by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;less resources&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Number of CPUs&amp;#039;&amp;#039;&amp;#039; requested by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;a lower number of CPUs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job Age&amp;#039;&amp;#039;&amp;#039;, i.e. the time that the job has been waiting in the queue&lt;br /&gt;
::: Used to provide higher priority to jobs which have been &amp;#039;&amp;#039;&amp;#039;queued for a longer time&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
::: Note: periods when a job is not eligible for running because of the [[#Limits on jobs by the same user|limits on running jobs by the same user]] do not increase job age.&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;FairShare&amp;#039;&amp;#039;&amp;#039;, i.e. a factor computed by SLURM to balance use of the system by different users&lt;br /&gt;
::: Used to provide higher priority to jobs by users who &amp;#039;&amp;#039;&amp;#039;used Mufasa&amp;#039;s resources less than others&amp;#039;&amp;#039;&amp;#039; (see [[#How FairShare works|below]])&lt;br /&gt;
&lt;br /&gt;
You can understand how each of these elements contributes to the priority of the jobs waiting for execution (both yours and of other users) with command &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;. A tailored version of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; that only displays the parameters listed above is the following:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sprio -o &amp;quot;%6i %9u %.8Y %.10B %.5Q %.9T %.5A %.5F&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Here is an example of the output of the command above:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
JOBID  USER      PRIORITY      ASSOC   QOS      TRES   AGE FAIRS&lt;br /&gt;
5790   amoglia       3495       2500   375  cpu=-800   933   487&lt;br /&gt;
5791   qliu          3010       2500   375  cpu=-800   916    19&lt;br /&gt;
5817   mzoccola      2336        625   750  cpu=-800   495  1266&lt;br /&gt;
5822   fbottazzi     2235        625   750  cpu=-600   467   994&lt;br /&gt;
5837   lpassoni      2508       2500   750  cpu=-800     0    58&lt;br /&gt;
5838   bagostini     4437        625  3000  cpu=-200     0  1013&lt;br /&gt;
5839   flazzaro      1549        625   750  cpu=-800     0   974&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
In it, column PRIORITY provides the overall priority of each job. Each priority value is the sum of the five values to its right; in particular, column ASSOC corresponds to user category, while column TRES corresponds to the number of CPUs. Note that the contribution of TRES to priority is &amp;#039;&amp;#039;negative&amp;#039;&amp;#039;, i.e. it is a penalty; its value is proportional to the number of CPUs requested by the job.&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;Things to consider when using &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
* Its output is only relevant for jobs &amp;#039;&amp;#039;not yet in execution&amp;#039;&amp;#039;. For an already running job, the values shown by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; correspond to the priority that the job would get if evaluated now: however, this priority does not affect job execution in any way.&lt;br /&gt;
* Jobs listed by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; may temporarily differ from jobs listed by &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;: this is due to the fact that SLURM updates its internal data (used by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;) every few minutes, while the output of &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt; is updated much more frequently.&lt;br /&gt;
&lt;br /&gt;
=== How FairShare works ===&lt;br /&gt;
In Mufasa 2.0, the FairShare of each user is computed according to these rules:&lt;br /&gt;
# It considers the quantity used by (current and past) user jobs of:&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;CPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of CPUs used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;RAM&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the amount of RAM used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;GPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of GPUs used&lt;br /&gt;
#** note: 40 GB GPUs have double the impact on FairShare than 20 GB GPUs&lt;br /&gt;
# The impact on FairShare of any use of resources is &amp;#039;&amp;#039;&amp;#039;proportional to its duration&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
#* example: using 32 GB of RAM for 48h has the same impact of using 64 GB of RAM for 24h&lt;br /&gt;
# FairShare has a &amp;quot;fading memory&amp;quot;: i.e., resource use has less and less impact on FairShare the farther in the past it is&lt;br /&gt;
#* consequence: over time, the &amp;quot;history&amp;quot; of a user gets forgotten by FairShare&lt;br /&gt;
&lt;br /&gt;
== How to maximise the priority of your jobs ==&lt;br /&gt;
&lt;br /&gt;
Every time you run a SLURM job, follow these guidelines:&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Choose the less powerful QOS compatible with the needs of your job&lt;br /&gt;
:: QOSes with access to less resources have higher base priority.&lt;br /&gt;
&lt;br /&gt;
; Only request CPUs and RAM that your job really needs&lt;br /&gt;
:: Usually, code exploits multiple CPUs only if specifically designed to do so: if unsure, only ask for 1 CPU.&lt;br /&gt;
:: The fewer resources your job asks for, the less it will wait before they become available.&lt;br /&gt;
:: The more CPUs your job asks for, the stronger the penalty to its priority.&lt;br /&gt;
:: Asking for fewer CPUs and/or less RAM improves your FairShare.&lt;br /&gt;
&lt;br /&gt;
; Do not request more time than your jobs needs to complete&lt;br /&gt;
:: Make a worst-case estimate and only ask for that duration.&lt;br /&gt;
:: Asking for less of Mufasa&amp;#039;s time improves your FairShare.&lt;br /&gt;
&lt;br /&gt;
; Debug and test code using less powerful QOSes before running it with more powerful QOSes&lt;br /&gt;
:: Your test jobs will get a higher priority and your FairShare will improve.&lt;br /&gt;
&lt;br /&gt;
; Cancel jobs when you don&amp;#039;t need them anymore&lt;br /&gt;
:: Always use [[User_Jobs#Cancelling_a_job_with_scancel|scancel]] to delete completed (or crashed) jobs: your Fairshare will improve.&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Protip: it&amp;#039;s a good idea to [[User Jobs#Looking for unused GPUs|check for unused GPUs]] before choosing what to request. Requesting a GPU that is currently idle will help your job get executed sooner.&lt;br /&gt;
&lt;br /&gt;
= SLURM partitions =&lt;br /&gt;
&lt;br /&gt;
Partitions are another mechanism provided by SLURM to create different levels of access to system resources. Since in Mufasa 2.0 access to resources is controlled via [[#SLURM Quality of Service (QOS)|QOSes]], partitions are not very relevant. (This is one of the main differences between Mufasa 1.0 and Mufasa 2.0.)&lt;br /&gt;
&lt;br /&gt;
Note, however, that the default values for some features of SLURM jobs (e.g., duration) are [[#Default values|set by the partition]].&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, there is a single SLURM partition, called &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt;, and all jobs run on it. The state of &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt; can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sinfo -o &amp;quot;%10P %5a %9T %11L %10l&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PARTITION  AVAIL STATE     DEFAULTTIME TIMELIMIT &lt;br /&gt;
jobs*      up    idle      1:00:00     3-00:00:00&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where columns correspond to the following information:&lt;br /&gt;
&lt;br /&gt;
:; PARTITION&lt;br /&gt;
:: name of the partition; the asterisks indicates that it&amp;#039;s the default one&lt;br /&gt;
&lt;br /&gt;
:; AVAIL&lt;br /&gt;
:: state/availability of the partition: see [[#Partition availability|below]]&lt;br /&gt;
&lt;br /&gt;
:; STATE&lt;br /&gt;
:: state (using [https://slurm.schedmd.com/sinfo.html#SECTION_NODE-STATE-CODES these codes])&lt;br /&gt;
:: typical values are &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;mixed&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that some of the resources are busy executing jobs while other are idle, and &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;allocated&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that all of the resources are in use&lt;br /&gt;
&lt;br /&gt;
:; DEFAULTTIME&lt;br /&gt;
:: default runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:; TIMELIMIT&lt;br /&gt;
:: maximum runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
The asterisk at the end of the partition name indicates the default partition, i.e. the one on which jobs which do not ask for a specific partition are run.&lt;br /&gt;
&lt;br /&gt;
Command &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; does not tell you about the &amp;#039;&amp;#039;jobs&amp;#039;&amp;#039; submitted to a partition. This information is obtained, instead, with [[User Jobs#Inspecting jobs with squeue|command &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;]].&lt;br /&gt;
&lt;br /&gt;
== Partition availability ==&lt;br /&gt;
&lt;br /&gt;
The most important information that &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; provides is the &amp;#039;&amp;#039;&amp;#039;availability&amp;#039;&amp;#039;&amp;#039; (also called &amp;#039;&amp;#039;&amp;#039;state&amp;#039;&amp;#039;&amp;#039;) of partitions. This is shown in column &amp;quot;AVAIL&amp;quot;. Possible partition states are:&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is available&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed as soon as resources allow&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is in the process of becoming unavailable (i.e., of entering the &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; state: see below)&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is unavailable&lt;br /&gt;
:: There are no running jobs&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
When a partition goes from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; no harm is done to running jobs. In a normally functioning SLURM system, the passage from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; happens only when no jobs are running on the partition. If (e.g., due to a malfunction) the passage happens with jobs still running, they get killed.&lt;br /&gt;
&lt;br /&gt;
A partition in state &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; requires intervention by a [[Roles|Job Administrator]] to be restored to &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
== Default values ==&lt;br /&gt;
&lt;br /&gt;
The features of SLURM partitions, including the &amp;#039;&amp;#039;&amp;#039;default values&amp;#039;&amp;#039;&amp;#039; which are applied to jobs that do not make explicit requests, can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scontrol show partition&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PartitionName=jobs&lt;br /&gt;
   AllowGroups=ALL AllowAccounts=ALL AllowQos=nogpu,gpulight,gpu,gpuwide,gpuheavy-20,gpuheavy-40&lt;br /&gt;
   AllocNodes=ALL Default=YES QoS=N/A&lt;br /&gt;
   DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO&lt;br /&gt;
   MaxNodes=UNLIMITED MaxTime=3-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED&lt;br /&gt;
   Nodes=gn01&lt;br /&gt;
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO&lt;br /&gt;
   OverTimeLimit=NONE PreemptMode=OFF&lt;br /&gt;
   State=UP TotalCPUs=48 TotalNodes=1 SelectTypeParameters=NONE&lt;br /&gt;
   JobDefaults=(null)&lt;br /&gt;
   DefMemPerNode=4096 MaxMemPerNode=UNLIMITED&lt;br /&gt;
   TRES=cpu=48,mem=1011435M,node=1,billing=49,gres/gpu=13,gres/gpu:3g.20gb=5,gres/gpu:40gb=3,gres/gpu:4g.20gb=5&lt;br /&gt;
   TRESBillingWeights=cpu=1.0,gres/gpu:3g.20gb=6.0,gres/gpu:4g.20gb=6.0,gres/gpu:40gb=6.0,mem=0.05g&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In the output of &amp;lt;code&amp;gt;scontrol show partition&amp;lt;/code&amp;gt; the most relevant default values for Mufasa users are:&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefaultTime&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default execution time assigned to a job run on the partition (e.g., 1 hour)&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefMemPerNode&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default amount of RAM assigned to a job run on the partition (e.g., 4GB)&lt;br /&gt;
&lt;br /&gt;
= System resources subjected to limitations =&lt;br /&gt;
&lt;br /&gt;
In systems based on SLURM like Mufasa, &amp;#039;&amp;#039;&amp;#039;TRES (Trackable RESources)&amp;#039;&amp;#039;&amp;#039; are (from [https://slurm.schedmd.com/tres.html SLURM&amp;#039;s documentation] &amp;quot;&amp;#039;&amp;#039;resources that can be tracked for usage or used to enforce limits against.&amp;#039;&amp;#039;&amp;quot;&lt;br /&gt;
&lt;br /&gt;
TRES include CPUs, RAM and &amp;#039;&amp;#039;&amp;#039;GRES&amp;#039;&amp;#039;&amp;#039;. The last term stands for &amp;#039;&amp;#039;Generic RESources&amp;#039;&amp;#039; that a job may need for its execution. In Mufasa, the only &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; resources are the GPUs.&lt;br /&gt;
&lt;br /&gt;
== &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax ==&lt;br /&gt;
&lt;br /&gt;
To ask SLURM to assign GRES resources (i.e., GPUs) to a job, a special syntax must be used. Precisely, the name of each GPU resource takes the form&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:Name:Type&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
Considering the [[System#CPUs and GPUs|GPU complement of Mufasa]], &amp;lt;code&amp;gt;Type&amp;lt;/code&amp;gt; takes the following values:&lt;br /&gt;
&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:40gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 40 Gbytes of RAM&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:4g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 4 compute units&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 3 compute units&lt;br /&gt;
&lt;br /&gt;
So, for instance,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
identifies a resource corresponding to a GPU with 20 GB of RAM and 3 compute units.&lt;br /&gt;
&lt;br /&gt;
When asking for a GRES resource (e.g., in an &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command or an &amp;lt;code&amp;gt;SBATCH&amp;lt;/code&amp;gt; directive of an [[User Jobs#Using execution scripts to run jobs|execution script]]), the syntax required by SLURM is&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:&amp;lt;Type&amp;gt;:&amp;lt;Quantity&amp;gt;&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
where &amp;lt;code&amp;gt;Quantity&amp;lt;/code&amp;gt; is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of type &amp;lt;code&amp;gt;4g.20gb&amp;lt;/code&amp;gt; the syntax is&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:4g.20gb:2&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SLURM&amp;#039;s &amp;#039;&amp;#039;generic resources&amp;#039;&amp;#039; are defined in &amp;lt;code&amp;gt;/etc/slurm/gres.conf&amp;lt;/code&amp;gt;. In order to make GPUs available to SLURM&amp;#039;s &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; management, Mufasa makes use of Nvidia&amp;#039;s [https://developer.nvidia.com/nvidia-management-library-nvml NVML library]. For additional information see [https://slurm.schedmd.com/gres.html SLURM&amp;#039;s documentation].&lt;/div&gt;</summary>
		<author><name>GiulioFontana</name></author>
	</entry>
	<entry>
		<id>https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2599</id>
		<title>SLURM</title>
		<link rel="alternate" type="text/html" href="https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2599"/>
		<updated>2026-11-18T22:35:07Z</updated>

		<summary type="html">&lt;p&gt;GiulioFontana: /* How to maximise the priority of your jobs */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page presents the features of SLURM that are most relevant to Mufasa&amp;#039;s [[Roles|Job Users]]. Job Users can submit jobs for execution, cancel their own jobs, and see other users&amp;#039; jobs (but not intervene on them).&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa &amp;#039;&amp;#039;&amp;#039;must use SLURM&amp;#039;&amp;#039;&amp;#039; to run resource-heavy processes, i.e. computing jobs that require one or more of the following:&lt;br /&gt;
* GPUs&lt;br /&gt;
* multiple CPUs&lt;br /&gt;
* powerful CPUs&lt;br /&gt;
* a significant amount of RAM&lt;br /&gt;
&lt;br /&gt;
In fact, only processes run via SLURM have access to all the resources of Mufasa. Processes run outside SLURM are executed by the [[System#Login server|login server]] virtual machine, which has minimal resources and no GPUs. Using SLURM is therefore the only way to execute resource-heavy jobs on Mufasa (this is a key difference between Mufasa 1.0 and Mufasa 2.0).&lt;br /&gt;
&lt;br /&gt;
= SLURM in a nutshell =&lt;br /&gt;
&lt;br /&gt;
Computation jobs on Mufasa needs to be launched via [[System#The SLURM job scheduling system|SLURM]]. SLURM provides jobs with access to the [[#System resources subjected to limitations|physical resources]] of Mufasa, such as CPUs, GPUs and RAM. Thanks to SLURM, processing jobs share system resources, optimising their occupation and availability. &lt;br /&gt;
&lt;br /&gt;
When a user runs a job, the job does not get executed immediately and is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039;. SLURM executes jobs according to their order in the queue: the top job in the queue gets executed as soon as the necessary resources are available, while jobs lower in the queue wait longer. The position of a job in the queue is due to the &amp;#039;&amp;#039;&amp;#039;[[#Job priority|priority]]&amp;#039;&amp;#039;&amp;#039; assigned to it by SLURM, with higher-priority jobs closer to the top. As a general rule:&lt;br /&gt;
&lt;br /&gt;
;: &amp;#039;&amp;#039;&amp;#039;the greater the fraction of Mufasa&amp;#039;s overall resources that a job asks for, the lower the job&amp;#039;s priority will be&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
The priority mechanism is used to encourage users to use Mufasa&amp;#039;s resources (i.e.: GPUs, CPUs, RAM, execution time) in an effective and equitable manner. This page includes a [[#How_to_maximise_the_priority_of_your_jobs|chart explaining how to maximise the priority of your jobs]].&lt;br /&gt;
&lt;br /&gt;
The &amp;#039;&amp;#039;&amp;#039;time&amp;#039;&amp;#039;&amp;#039; available to a job for its execution is controlled by SLURM. When a user requests execution of a job, they must specify the duration of the time slot that the job needs. The job must complete its execution before the end of the requested time slot, because at the end of such slot it gets killed by SLURM.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0 access to system resources is managed via SLURM&amp;#039;s [[#SLURM Quality of Service (QOS)|Quality of Service (QOS)]] mechanism (Mufasa 1.0 used [[#SLURM_partitions|partitions]] instead). While [[User Jobs#Running jobs with SLURM|launching a processing job via SLURM]], the user must always specify the chosen QOS. QOSes differ in the set of resources that they provide access to because each of them is designed to fit a given type of job.&lt;br /&gt;
&lt;br /&gt;
Mufasa sets limits to the number of jobs by the same user. This page includes a [[#Limits on jobs by the same user|table summarising such limits]].&lt;br /&gt;
&lt;br /&gt;
= SLURM Quality of Service (QOS) =&lt;br /&gt;
&lt;br /&gt;
Through &amp;#039;&amp;#039;&amp;#039;Quality of Services&amp;#039;&amp;#039;&amp;#039; (&amp;#039;&amp;#039;&amp;#039;QOSes&amp;#039;&amp;#039;&amp;#039;), SLURM lets system configurators assign a name to a set of related constraints.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, QOSes are used to define different levels of access to the server&amp;#039;s resources. When [[User Jobs|executing a job with SLURM]], a user must always &amp;#039;&amp;#039;&amp;#039;specify the QOS&amp;#039;&amp;#039;&amp;#039; that their job will use: this choice, in turn, determines what resources the job is able to access and influences the [[#Job priority|priority]] of the job.&lt;br /&gt;
&lt;br /&gt;
Mufasa&amp;#039;s QOSes and their features can be inspected with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list qos format=name%-11,priority,MaxSubmit,maxwall,maxtres%-80,mintres%-18&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following (see below for the same data in table form):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
Name          Priority MaxSubmit     MaxWall MaxTRES                                                                          MinTRES            &lt;br /&gt;
----------- ---------- --------- ----------- -------------------------------------------------------------------------------- ------------------ &lt;br /&gt;
normal               0                                                                                                                           &lt;br /&gt;
nogpu                4         1  3-00:00:00 cpu=16,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=128G                    &lt;br /&gt;
gpuheavy-20          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=2,mem=128G             gres/gpu:4g.20gb=1 &lt;br /&gt;
gpuheavy-40          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=1,gres/gpu:4g.20gb=0,mem=128G             gres/gpu:40gb=1    &lt;br /&gt;
gpulight             8         1    12:00:00 cpu=2,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpu                  2         1  1-00:00:00 cpu=8,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpuwide              2         2  1-00:00:00 cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=1,mem=64G              gres/gpu:4g.20gb=1 &lt;br /&gt;
build               32         1    02:00:00 cpu=2,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=16G&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The columns of this output are the following:&lt;br /&gt;
&lt;br /&gt;
:; Name&lt;br /&gt;
:: name of the QOS&lt;br /&gt;
&lt;br /&gt;
:; Priority&lt;br /&gt;
:: priority tier associated to the QOS (higher value = higher priority): see [[#Job priority|Job priority]] for details&lt;br /&gt;
&lt;br /&gt;
:; MaxSubmit&lt;br /&gt;
:: maximum number of jobs from a single user that can be submitted to SLURM with this QOS; submitted jobs include both running and queued jobs&lt;br /&gt;
:: See [[#Limits on jobs by the same user|Limits on jobs by the same user]] for an overview of the limits on jobs set by Mufasa.&lt;br /&gt;
&lt;br /&gt;
:; MaxWall&lt;br /&gt;
:: maximum wall clock duration of the jobs using the QOS (after which they are killed by SLURM), in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
:: For some QOSes these are not set: it means that they are determined by the [[#SLURM partitions|partition]]. Partitions also define the [[#Default values|default duration]] of jobs.&lt;br /&gt;
&lt;br /&gt;
:; MaxTRES&lt;br /&gt;
:: amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) available to a job using the QOS, where&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;cpu=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of CPUs (i.e., processor cores) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of CPUs specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;gres/&amp;#039;&amp;#039;gpu:Type&amp;#039;&amp;#039;=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of GPUs of class &amp;lt;code&amp;gt;&amp;#039;&amp;#039;Type&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; (see [[User Jobs#gres syntax|&amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax]]) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; (for QOSes that allow access to GPUs) if not specified, the job cannot be launched&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;mem=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;G&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum amount of system RAM is &amp;#039;&amp;#039;K&amp;#039;&amp;#039; GBytes&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of RAM specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
&lt;br /&gt;
:; MinTRES&lt;br /&gt;
:: minimum amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) that a job using the QOS must request in order to actually get executed by SLURM.&lt;br /&gt;
:: (If your job does not actually need these resources, you&amp;#039;ve chosen the wrong QOS: you can use one with a higher priority.)&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; QOS is the default one, and exists only to ensure that users always specify a QOS when running a job. Since &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; has zero priority and no resources, a job run using this QOS would never be run.&lt;br /&gt;
&lt;br /&gt;
The information provided by the &amp;lt;code&amp;gt;sacctmgr list qos&amp;lt;/code&amp;gt; command above is summarised by the following table:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! QOS&lt;br /&gt;
! Priority&amp;lt;/br&amp;gt;tier&lt;br /&gt;
! Max&amp;lt;/br&amp;gt;Submit&lt;br /&gt;
! MaxWall&amp;lt;/br&amp;gt;[h]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;CPUs&lt;br /&gt;
! max RAM&amp;lt;/br&amp;gt;[GB]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;3g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;4g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;40GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | build&lt;br /&gt;
| 32&lt;br /&gt;
| 1&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 16&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpulight&lt;br /&gt;
| 8&lt;br /&gt;
| 1&lt;br /&gt;
| 12&lt;br /&gt;
| 2&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | nogpu&lt;br /&gt;
| 4&lt;br /&gt;
| 1&lt;br /&gt;
| 72&lt;br /&gt;
| 16&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpu&lt;br /&gt;
| 2&lt;br /&gt;
| 1&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuwide&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-20&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| 2 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-40&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
A key piece of information in the table above is the &amp;#039;&amp;#039;&amp;#039;priority tier&amp;#039;&amp;#039;&amp;#039; associated to each QOS. In Mufasa 2.0, priority tiers are used to encourage users to use the &amp;#039;&amp;#039;&amp;#039;least powerful QOS that is compatible with their needs&amp;#039;&amp;#039;&amp;#039;, where &amp;quot;powerful&amp;quot; means &amp;quot;rich with resources&amp;quot;. Less powerful QOSes increase the priority of the jobs that use them, so these jobs tend to be executed sooner. See [[#Job priority|Job priority]] for details about how priority affects the execution order of jobs in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
== The &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS ==&lt;br /&gt;
&lt;br /&gt;
This QOS is specifically designed to be used by Mufasa users to &amp;#039;&amp;#039;&amp;#039;quickly build [[System#Containers|container images]]&amp;#039;&amp;#039;&amp;#039;. Its associated priority tier is very high: this, combined with the fact that jobs using this QOS require few resources, means that such jobs usually get executed very soon.&lt;br /&gt;
&lt;br /&gt;
On the other hand, the limited resources and the lack of access to the GPUs make the &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS unsuitable for other tasks.&lt;br /&gt;
&lt;br /&gt;
See [[Singularity#Building Singularity images|Building Singularity images]] for directions about building Singularity container images.&lt;br /&gt;
&lt;br /&gt;
== Restricted QOSes ==&lt;br /&gt;
&lt;br /&gt;
In Mufasa, the most powerful QOSes are reserved to researchers (called &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users: these include academic personnel and Ph.D. students). M.Sc. students, called &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users, cannot use them while running jobs.&lt;br /&gt;
&lt;br /&gt;
See [[#research users and students users|below]] to understand how &amp;#039;&amp;#039;user categories&amp;#039;&amp;#039; work in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
= &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users =&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa belong to two &amp;#039;&amp;#039;&amp;#039;user categories&amp;#039;&amp;#039;&amp;#039;. Categories provide users with different access to  to Mufasa&amp;#039;s resources: the idea being to provide researchers with more access while still letting students benefit from the server&amp;#039;s capabilities.&lt;br /&gt;
&lt;br /&gt;
User categories in Mufasa are the following:&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. academic personnel and Ph.D. students&lt;br /&gt;
::: * can use all QOSes, including the the [[#Restricted QOSes|restricted ones]]&lt;br /&gt;
::: * their jobs have a higher &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a higher number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. M.Sc. students&lt;br /&gt;
::: * cannot use the [[#Restricted QOSes|restricted QOSes]]&lt;br /&gt;
::: * their jobs have a lower &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a lower number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
You can inspect the differences in priority and running jobs between &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association format=&amp;quot;account,priority,maxjobs&amp;quot; | grep -E &amp;#039;Account|research|students&amp;#039;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
   Account   Priority MaxJobs &lt;br /&gt;
  research          4       2 &lt;br /&gt;
  students          1       1&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
To know what limits apply to your own user, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association where user=$USER format=&amp;quot;user,priority,maxjobs,qos%-60&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
      User   Priority MaxJobs QOS                                                          &lt;br /&gt;
---------- ---------- ------- ------------------------------------------------------------ &lt;br /&gt;
    preali          4       2 build,gpu,gpuheavy-20,gpuheavy-40,gpulight,gpuwide,nogpu&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The list under &amp;quot;QOS&amp;quot; shows what QOSes your user is allowed to use when [[User Jobs|running jobs]]. &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users can use all of them, while &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users can only access a subset of them.&lt;br /&gt;
&lt;br /&gt;
= Limits on jobs by the same user =&lt;br /&gt;
&lt;br /&gt;
Mufasa enforces limits on the number of jobs from a single user. Such limits aim at preventing users from &amp;quot;hogging&amp;quot; system resources, and apply to:&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;submitted jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that the user asked SLURM to execute, each of which may currently be either running or queued&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;running jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that are currently in execution&lt;br /&gt;
&lt;br /&gt;
Limits on submitted jobs are set via [[#SLURM Quality of Service (QOS)|QOSes]], while limits on running jobs are set via [[#research users and students users|user category]].&lt;br /&gt;
&lt;br /&gt;
The following table summarises what limits exist:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! &lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;running&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;submitted&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | global limits&amp;lt;br/&amp;gt;(system-wide)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to limits set by individual QOSes (below)&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | limits for&amp;lt;br/&amp;gt;individual&amp;lt;/br&amp;gt;QOSes&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to global limits (above)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for the&amp;#039;&amp;#039; &amp;lt;code&amp;gt;gpuwide&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;QOS&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for each of the other QOSes&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
= Job priority =&lt;br /&gt;
&lt;br /&gt;
Once the execution of a job has been requested, the job is not run immediately: it is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039; by SLURM, together with all the other jobs awaiting execution. The job on top of the queue at any time is the first to be put into execution as soon as the resources it requires are available. &lt;br /&gt;
&lt;br /&gt;
The order of the items in the job queue depends on their &amp;#039;&amp;#039;&amp;#039;priority&amp;#039;&amp;#039;&amp;#039;. The job with the highest priority is put by SLURM on top of the queue, while all other jobs are put in the queue in order of descending priority.&lt;br /&gt;
&lt;br /&gt;
The goal of SLURM is to maximise resource availability: i.e., to ensure the shorter possible wait time before job execution. To achieve this goal, in Mufasa SLURM is configured to &amp;#039;&amp;#039;&amp;#039;encourage users not to ask for resources or execution time that their job doesn&amp;#039;t need&amp;#039;&amp;#039;&amp;#039;. This is done via the priority mechanism: the more resources and/or time a job requests, the lower its priority will be; and the later it will be executed.&lt;br /&gt;
&lt;br /&gt;
Priority management in Mufasa is designed to set up a &amp;#039;&amp;#039;&amp;#039;virtuous cycle&amp;#039;&amp;#039;&amp;#039; where users, by carefully choosing what to ask for, obtain two results:&lt;br /&gt;
* they ensure that their job is executed as soon as possible;&lt;br /&gt;
* they leave as much as possible of Mufasa&amp;#039;s resources free for other users&amp;#039;s jobs.&lt;br /&gt;
&lt;br /&gt;
== Elements determining job priority ==&lt;br /&gt;
In Mufasa, the priority of a job is computed by SLURM according to the following elements:&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#research users and students users|User category]]&amp;#039;&amp;#039;&amp;#039; (i.e., &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;)&lt;br /&gt;
::: Used to provide higher priority to jobs run by &amp;#039;&amp;#039;&amp;#039;research personnel&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#SLURM Quality of Service (QOS)|QOS]]&amp;#039;&amp;#039;&amp;#039; used by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;less resources&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Number of CPUs&amp;#039;&amp;#039;&amp;#039; requested by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;a lower number of CPUs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job Age&amp;#039;&amp;#039;&amp;#039;, i.e. the time that the job has been waiting in the queue&lt;br /&gt;
::: Used to provide higher priority to jobs which have been &amp;#039;&amp;#039;&amp;#039;queued for a longer time&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
::: Note: periods when a job is not eligible for running because of the [[#Limits on jobs by the same user|limits on running jobs by the same user]] do not increase job age.&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;FairShare&amp;#039;&amp;#039;&amp;#039;, i.e. a factor computed by SLURM to balance use of the system by different users&lt;br /&gt;
::: Used to provide higher priority to jobs by users who &amp;#039;&amp;#039;&amp;#039;used Mufasa&amp;#039;s resources less than others&amp;#039;&amp;#039;&amp;#039; (see [[#How FairShare works|below]])&lt;br /&gt;
&lt;br /&gt;
You can understand how each of these elements contributes to the priority of the jobs waiting for execution (both yours and of other users) with command &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;. A tailored version of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; that only displays the parameters listed above is the following:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sprio -o &amp;quot;%6i %9u %.8Y %.10B %.5Q %.9T %.5A %.5F&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Here is an example of the output of the command above:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
JOBID  USER      PRIORITY      ASSOC   QOS      TRES   AGE FAIRS&lt;br /&gt;
5790   amoglia       3495       2500   375  cpu=-800   933   487&lt;br /&gt;
5791   qliu          3010       2500   375  cpu=-800   916    19&lt;br /&gt;
5817   mzoccola      2336        625   750  cpu=-800   495  1266&lt;br /&gt;
5822   fbottazzi     2235        625   750  cpu=-600   467   994&lt;br /&gt;
5837   lpassoni      2508       2500   750  cpu=-800     0    58&lt;br /&gt;
5838   bagostini     4437        625  3000  cpu=-200     0  1013&lt;br /&gt;
5839   flazzaro      1549        625   750  cpu=-800     0   974&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
In it, column PRIORITY provides the overall priority of each job. Each priority value is the sum of the five values to its right; in particular, column ASSOC corresponds to user category, while column TRES corresponds to the number of CPUs. Note that the contribution of TRES to priority is &amp;#039;&amp;#039;negative&amp;#039;&amp;#039;, i.e. it is a penalty; its value is proportional to the number of CPUs requested by the job.&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;Things to consider when using &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
* Its output is only relevant for jobs &amp;#039;&amp;#039;not yet in execution&amp;#039;&amp;#039;. For an already running job, the values shown by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; correspond to the priority that the job would get if evaluated now: however, this priority does not affect job execution in any way.&lt;br /&gt;
* Jobs listed by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; may temporarily differ from jobs listed by &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;: this is due to the fact that SLURM updates its internal data (used by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;) every few minutes, while the output of &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt; is updated much more frequently.&lt;br /&gt;
&lt;br /&gt;
=== How FairShare works ===&lt;br /&gt;
In Mufasa 2.0, the FairShare of each user is computed according to these rules:&lt;br /&gt;
# It considers the quantity used by (current and past) user jobs of:&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;CPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of CPUs used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;RAM&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the amount of RAM used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;GPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of GPUs used&lt;br /&gt;
#** note: 40 GB GPUs have double the impact on FairShare than 20 GB GPUs&lt;br /&gt;
# The impact on FairShare of any use of resources is &amp;#039;&amp;#039;&amp;#039;proportional to its duration&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
#* example: using 32 GB of RAM for 48h has the same impact of using 64 GB of RAM for 24h&lt;br /&gt;
# FairShare has a &amp;quot;fading memory&amp;quot;: i.e., resource use has less and less impact on FairShare the farther in the past it is&lt;br /&gt;
#* consequence: over time, the &amp;quot;history&amp;quot; of a user gets forgotten by FairShare&lt;br /&gt;
&lt;br /&gt;
== How to maximise the priority of your jobs ==&lt;br /&gt;
&lt;br /&gt;
Every time you run a SLURM job, follow these guidelines:&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Choose the less powerful QOS compatible with the needs of your job&lt;br /&gt;
:: QOSes with access to less resources have higher base priority.&lt;br /&gt;
&lt;br /&gt;
; Only request CPUs and RAM that your job really needs&lt;br /&gt;
:: Usually, code exploits multiple CPUs only if specifically designed to do so: if unsure, only ask for 1 CPU.&lt;br /&gt;
:: The fewer resources your job asks for, the less it will wait before they become available.&lt;br /&gt;
:: The more CPUs your job asks for, the stronger the penalty to its priority.&lt;br /&gt;
:: Asking for fewer CPUs and/or less RAM improves your FairShare.&lt;br /&gt;
&lt;br /&gt;
; Do not request more time than your jobs needs to complete&lt;br /&gt;
:: Make a worst-case estimate and only ask for that duration.&lt;br /&gt;
:: Asking for less of Mufasa&amp;#039;s time improves your FairShare.&lt;br /&gt;
&lt;br /&gt;
; Debug and test code using less powerful QOSes before running it with more powerful QOSes&lt;br /&gt;
:: Your test jobs will get a higher priority and your FairShare will improve.&lt;br /&gt;
&lt;br /&gt;
; Cancel jobs that you don&amp;#039;t need them anymore&lt;br /&gt;
:: Always use [[User_Jobs#Cancelling_a_job_with_scancel|scancel]] to delete completed (or crashed) jobs: your Fairshare will improve.&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Protip: it&amp;#039;s a good idea to [[User Jobs#Looking for unused GPUs|check for unused GPUs]] before choosing what to request. Requesting a GPU that is currently idle will help your job get executed sooner.&lt;br /&gt;
&lt;br /&gt;
= SLURM partitions =&lt;br /&gt;
&lt;br /&gt;
Partitions are another mechanism provided by SLURM to create different levels of access to system resources. Since in Mufasa 2.0 access to resources is controlled via [[#SLURM Quality of Service (QOS)|QOSes]], partitions are not very relevant. (This is one of the main differences between Mufasa 1.0 and Mufasa 2.0.)&lt;br /&gt;
&lt;br /&gt;
Note, however, that the default values for some features of SLURM jobs (e.g., duration) are [[#Default values|set by the partition]].&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, there is a single SLURM partition, called &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt;, and all jobs run on it. The state of &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt; can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sinfo -o &amp;quot;%10P %5a %9T %11L %10l&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PARTITION  AVAIL STATE     DEFAULTTIME TIMELIMIT &lt;br /&gt;
jobs*      up    idle      1:00:00     3-00:00:00&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where columns correspond to the following information:&lt;br /&gt;
&lt;br /&gt;
:; PARTITION&lt;br /&gt;
:: name of the partition; the asterisks indicates that it&amp;#039;s the default one&lt;br /&gt;
&lt;br /&gt;
:; AVAIL&lt;br /&gt;
:: state/availability of the partition: see [[#Partition availability|below]]&lt;br /&gt;
&lt;br /&gt;
:; STATE&lt;br /&gt;
:: state (using [https://slurm.schedmd.com/sinfo.html#SECTION_NODE-STATE-CODES these codes])&lt;br /&gt;
:: typical values are &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;mixed&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that some of the resources are busy executing jobs while other are idle, and &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;allocated&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that all of the resources are in use&lt;br /&gt;
&lt;br /&gt;
:; DEFAULTTIME&lt;br /&gt;
:: default runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:; TIMELIMIT&lt;br /&gt;
:: maximum runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
The asterisk at the end of the partition name indicates the default partition, i.e. the one on which jobs which do not ask for a specific partition are run.&lt;br /&gt;
&lt;br /&gt;
Command &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; does not tell you about the &amp;#039;&amp;#039;jobs&amp;#039;&amp;#039; submitted to a partition. This information is obtained, instead, with [[User Jobs#Inspecting jobs with squeue|command &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;]].&lt;br /&gt;
&lt;br /&gt;
== Partition availability ==&lt;br /&gt;
&lt;br /&gt;
The most important information that &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; provides is the &amp;#039;&amp;#039;&amp;#039;availability&amp;#039;&amp;#039;&amp;#039; (also called &amp;#039;&amp;#039;&amp;#039;state&amp;#039;&amp;#039;&amp;#039;) of partitions. This is shown in column &amp;quot;AVAIL&amp;quot;. Possible partition states are:&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is available&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed as soon as resources allow&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is in the process of becoming unavailable (i.e., of entering the &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; state: see below)&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is unavailable&lt;br /&gt;
:: There are no running jobs&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
When a partition goes from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; no harm is done to running jobs. In a normally functioning SLURM system, the passage from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; happens only when no jobs are running on the partition. If (e.g., due to a malfunction) the passage happens with jobs still running, they get killed.&lt;br /&gt;
&lt;br /&gt;
A partition in state &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; requires intervention by a [[Roles|Job Administrator]] to be restored to &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
== Default values ==&lt;br /&gt;
&lt;br /&gt;
The features of SLURM partitions, including the &amp;#039;&amp;#039;&amp;#039;default values&amp;#039;&amp;#039;&amp;#039; which are applied to jobs that do not make explicit requests, can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scontrol show partition&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PartitionName=jobs&lt;br /&gt;
   AllowGroups=ALL AllowAccounts=ALL AllowQos=nogpu,gpulight,gpu,gpuwide,gpuheavy-20,gpuheavy-40&lt;br /&gt;
   AllocNodes=ALL Default=YES QoS=N/A&lt;br /&gt;
   DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO&lt;br /&gt;
   MaxNodes=UNLIMITED MaxTime=3-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED&lt;br /&gt;
   Nodes=gn01&lt;br /&gt;
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO&lt;br /&gt;
   OverTimeLimit=NONE PreemptMode=OFF&lt;br /&gt;
   State=UP TotalCPUs=48 TotalNodes=1 SelectTypeParameters=NONE&lt;br /&gt;
   JobDefaults=(null)&lt;br /&gt;
   DefMemPerNode=4096 MaxMemPerNode=UNLIMITED&lt;br /&gt;
   TRES=cpu=48,mem=1011435M,node=1,billing=49,gres/gpu=13,gres/gpu:3g.20gb=5,gres/gpu:40gb=3,gres/gpu:4g.20gb=5&lt;br /&gt;
   TRESBillingWeights=cpu=1.0,gres/gpu:3g.20gb=6.0,gres/gpu:4g.20gb=6.0,gres/gpu:40gb=6.0,mem=0.05g&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In the output of &amp;lt;code&amp;gt;scontrol show partition&amp;lt;/code&amp;gt; the most relevant default values for Mufasa users are:&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefaultTime&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default execution time assigned to a job run on the partition (e.g., 1 hour)&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefMemPerNode&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default amount of RAM assigned to a job run on the partition (e.g., 4GB)&lt;br /&gt;
&lt;br /&gt;
= System resources subjected to limitations =&lt;br /&gt;
&lt;br /&gt;
In systems based on SLURM like Mufasa, &amp;#039;&amp;#039;&amp;#039;TRES (Trackable RESources)&amp;#039;&amp;#039;&amp;#039; are (from [https://slurm.schedmd.com/tres.html SLURM&amp;#039;s documentation] &amp;quot;&amp;#039;&amp;#039;resources that can be tracked for usage or used to enforce limits against.&amp;#039;&amp;#039;&amp;quot;&lt;br /&gt;
&lt;br /&gt;
TRES include CPUs, RAM and &amp;#039;&amp;#039;&amp;#039;GRES&amp;#039;&amp;#039;&amp;#039;. The last term stands for &amp;#039;&amp;#039;Generic RESources&amp;#039;&amp;#039; that a job may need for its execution. In Mufasa, the only &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; resources are the GPUs.&lt;br /&gt;
&lt;br /&gt;
== &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax ==&lt;br /&gt;
&lt;br /&gt;
To ask SLURM to assign GRES resources (i.e., GPUs) to a job, a special syntax must be used. Precisely, the name of each GPU resource takes the form&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:Name:Type&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
Considering the [[System#CPUs and GPUs|GPU complement of Mufasa]], &amp;lt;code&amp;gt;Type&amp;lt;/code&amp;gt; takes the following values:&lt;br /&gt;
&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:40gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 40 Gbytes of RAM&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:4g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 4 compute units&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 3 compute units&lt;br /&gt;
&lt;br /&gt;
So, for instance,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
identifies a resource corresponding to a GPU with 20 GB of RAM and 3 compute units.&lt;br /&gt;
&lt;br /&gt;
When asking for a GRES resource (e.g., in an &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command or an &amp;lt;code&amp;gt;SBATCH&amp;lt;/code&amp;gt; directive of an [[User Jobs#Using execution scripts to run jobs|execution script]]), the syntax required by SLURM is&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:&amp;lt;Type&amp;gt;:&amp;lt;Quantity&amp;gt;&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
where &amp;lt;code&amp;gt;Quantity&amp;lt;/code&amp;gt; is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of type &amp;lt;code&amp;gt;4g.20gb&amp;lt;/code&amp;gt; the syntax is&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:4g.20gb:2&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SLURM&amp;#039;s &amp;#039;&amp;#039;generic resources&amp;#039;&amp;#039; are defined in &amp;lt;code&amp;gt;/etc/slurm/gres.conf&amp;lt;/code&amp;gt;. In order to make GPUs available to SLURM&amp;#039;s &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; management, Mufasa makes use of Nvidia&amp;#039;s [https://developer.nvidia.com/nvidia-management-library-nvml NVML library]. For additional information see [https://slurm.schedmd.com/gres.html SLURM&amp;#039;s documentation].&lt;/div&gt;</summary>
		<author><name>GiulioFontana</name></author>
	</entry>
	<entry>
		<id>https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2598</id>
		<title>SLURM</title>
		<link rel="alternate" type="text/html" href="https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2598"/>
		<updated>2026-11-18T22:32:07Z</updated>

		<summary type="html">&lt;p&gt;GiulioFontana: /* Elements determining job priority */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page presents the features of SLURM that are most relevant to Mufasa&amp;#039;s [[Roles|Job Users]]. Job Users can submit jobs for execution, cancel their own jobs, and see other users&amp;#039; jobs (but not intervene on them).&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa &amp;#039;&amp;#039;&amp;#039;must use SLURM&amp;#039;&amp;#039;&amp;#039; to run resource-heavy processes, i.e. computing jobs that require one or more of the following:&lt;br /&gt;
* GPUs&lt;br /&gt;
* multiple CPUs&lt;br /&gt;
* powerful CPUs&lt;br /&gt;
* a significant amount of RAM&lt;br /&gt;
&lt;br /&gt;
In fact, only processes run via SLURM have access to all the resources of Mufasa. Processes run outside SLURM are executed by the [[System#Login server|login server]] virtual machine, which has minimal resources and no GPUs. Using SLURM is therefore the only way to execute resource-heavy jobs on Mufasa (this is a key difference between Mufasa 1.0 and Mufasa 2.0).&lt;br /&gt;
&lt;br /&gt;
= SLURM in a nutshell =&lt;br /&gt;
&lt;br /&gt;
Computation jobs on Mufasa needs to be launched via [[System#The SLURM job scheduling system|SLURM]]. SLURM provides jobs with access to the [[#System resources subjected to limitations|physical resources]] of Mufasa, such as CPUs, GPUs and RAM. Thanks to SLURM, processing jobs share system resources, optimising their occupation and availability. &lt;br /&gt;
&lt;br /&gt;
When a user runs a job, the job does not get executed immediately and is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039;. SLURM executes jobs according to their order in the queue: the top job in the queue gets executed as soon as the necessary resources are available, while jobs lower in the queue wait longer. The position of a job in the queue is due to the &amp;#039;&amp;#039;&amp;#039;[[#Job priority|priority]]&amp;#039;&amp;#039;&amp;#039; assigned to it by SLURM, with higher-priority jobs closer to the top. As a general rule:&lt;br /&gt;
&lt;br /&gt;
;: &amp;#039;&amp;#039;&amp;#039;the greater the fraction of Mufasa&amp;#039;s overall resources that a job asks for, the lower the job&amp;#039;s priority will be&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
The priority mechanism is used to encourage users to use Mufasa&amp;#039;s resources (i.e.: GPUs, CPUs, RAM, execution time) in an effective and equitable manner. This page includes a [[#How_to_maximise_the_priority_of_your_jobs|chart explaining how to maximise the priority of your jobs]].&lt;br /&gt;
&lt;br /&gt;
The &amp;#039;&amp;#039;&amp;#039;time&amp;#039;&amp;#039;&amp;#039; available to a job for its execution is controlled by SLURM. When a user requests execution of a job, they must specify the duration of the time slot that the job needs. The job must complete its execution before the end of the requested time slot, because at the end of such slot it gets killed by SLURM.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0 access to system resources is managed via SLURM&amp;#039;s [[#SLURM Quality of Service (QOS)|Quality of Service (QOS)]] mechanism (Mufasa 1.0 used [[#SLURM_partitions|partitions]] instead). While [[User Jobs#Running jobs with SLURM|launching a processing job via SLURM]], the user must always specify the chosen QOS. QOSes differ in the set of resources that they provide access to because each of them is designed to fit a given type of job.&lt;br /&gt;
&lt;br /&gt;
Mufasa sets limits to the number of jobs by the same user. This page includes a [[#Limits on jobs by the same user|table summarising such limits]].&lt;br /&gt;
&lt;br /&gt;
= SLURM Quality of Service (QOS) =&lt;br /&gt;
&lt;br /&gt;
Through &amp;#039;&amp;#039;&amp;#039;Quality of Services&amp;#039;&amp;#039;&amp;#039; (&amp;#039;&amp;#039;&amp;#039;QOSes&amp;#039;&amp;#039;&amp;#039;), SLURM lets system configurators assign a name to a set of related constraints.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, QOSes are used to define different levels of access to the server&amp;#039;s resources. When [[User Jobs|executing a job with SLURM]], a user must always &amp;#039;&amp;#039;&amp;#039;specify the QOS&amp;#039;&amp;#039;&amp;#039; that their job will use: this choice, in turn, determines what resources the job is able to access and influences the [[#Job priority|priority]] of the job.&lt;br /&gt;
&lt;br /&gt;
Mufasa&amp;#039;s QOSes and their features can be inspected with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list qos format=name%-11,priority,MaxSubmit,maxwall,maxtres%-80,mintres%-18&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following (see below for the same data in table form):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
Name          Priority MaxSubmit     MaxWall MaxTRES                                                                          MinTRES            &lt;br /&gt;
----------- ---------- --------- ----------- -------------------------------------------------------------------------------- ------------------ &lt;br /&gt;
normal               0                                                                                                                           &lt;br /&gt;
nogpu                4         1  3-00:00:00 cpu=16,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=128G                    &lt;br /&gt;
gpuheavy-20          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=2,mem=128G             gres/gpu:4g.20gb=1 &lt;br /&gt;
gpuheavy-40          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=1,gres/gpu:4g.20gb=0,mem=128G             gres/gpu:40gb=1    &lt;br /&gt;
gpulight             8         1    12:00:00 cpu=2,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpu                  2         1  1-00:00:00 cpu=8,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpuwide              2         2  1-00:00:00 cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=1,mem=64G              gres/gpu:4g.20gb=1 &lt;br /&gt;
build               32         1    02:00:00 cpu=2,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=16G&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The columns of this output are the following:&lt;br /&gt;
&lt;br /&gt;
:; Name&lt;br /&gt;
:: name of the QOS&lt;br /&gt;
&lt;br /&gt;
:; Priority&lt;br /&gt;
:: priority tier associated to the QOS (higher value = higher priority): see [[#Job priority|Job priority]] for details&lt;br /&gt;
&lt;br /&gt;
:; MaxSubmit&lt;br /&gt;
:: maximum number of jobs from a single user that can be submitted to SLURM with this QOS; submitted jobs include both running and queued jobs&lt;br /&gt;
:: See [[#Limits on jobs by the same user|Limits on jobs by the same user]] for an overview of the limits on jobs set by Mufasa.&lt;br /&gt;
&lt;br /&gt;
:; MaxWall&lt;br /&gt;
:: maximum wall clock duration of the jobs using the QOS (after which they are killed by SLURM), in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
:: For some QOSes these are not set: it means that they are determined by the [[#SLURM partitions|partition]]. Partitions also define the [[#Default values|default duration]] of jobs.&lt;br /&gt;
&lt;br /&gt;
:; MaxTRES&lt;br /&gt;
:: amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) available to a job using the QOS, where&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;cpu=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of CPUs (i.e., processor cores) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of CPUs specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;gres/&amp;#039;&amp;#039;gpu:Type&amp;#039;&amp;#039;=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of GPUs of class &amp;lt;code&amp;gt;&amp;#039;&amp;#039;Type&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; (see [[User Jobs#gres syntax|&amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax]]) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; (for QOSes that allow access to GPUs) if not specified, the job cannot be launched&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;mem=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;G&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum amount of system RAM is &amp;#039;&amp;#039;K&amp;#039;&amp;#039; GBytes&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of RAM specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
&lt;br /&gt;
:; MinTRES&lt;br /&gt;
:: minimum amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) that a job using the QOS must request in order to actually get executed by SLURM.&lt;br /&gt;
:: (If your job does not actually need these resources, you&amp;#039;ve chosen the wrong QOS: you can use one with a higher priority.)&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; QOS is the default one, and exists only to ensure that users always specify a QOS when running a job. Since &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; has zero priority and no resources, a job run using this QOS would never be run.&lt;br /&gt;
&lt;br /&gt;
The information provided by the &amp;lt;code&amp;gt;sacctmgr list qos&amp;lt;/code&amp;gt; command above is summarised by the following table:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! QOS&lt;br /&gt;
! Priority&amp;lt;/br&amp;gt;tier&lt;br /&gt;
! Max&amp;lt;/br&amp;gt;Submit&lt;br /&gt;
! MaxWall&amp;lt;/br&amp;gt;[h]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;CPUs&lt;br /&gt;
! max RAM&amp;lt;/br&amp;gt;[GB]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;3g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;4g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;40GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | build&lt;br /&gt;
| 32&lt;br /&gt;
| 1&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 16&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpulight&lt;br /&gt;
| 8&lt;br /&gt;
| 1&lt;br /&gt;
| 12&lt;br /&gt;
| 2&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | nogpu&lt;br /&gt;
| 4&lt;br /&gt;
| 1&lt;br /&gt;
| 72&lt;br /&gt;
| 16&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpu&lt;br /&gt;
| 2&lt;br /&gt;
| 1&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuwide&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-20&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| 2 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-40&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
A key piece of information in the table above is the &amp;#039;&amp;#039;&amp;#039;priority tier&amp;#039;&amp;#039;&amp;#039; associated to each QOS. In Mufasa 2.0, priority tiers are used to encourage users to use the &amp;#039;&amp;#039;&amp;#039;least powerful QOS that is compatible with their needs&amp;#039;&amp;#039;&amp;#039;, where &amp;quot;powerful&amp;quot; means &amp;quot;rich with resources&amp;quot;. Less powerful QOSes increase the priority of the jobs that use them, so these jobs tend to be executed sooner. See [[#Job priority|Job priority]] for details about how priority affects the execution order of jobs in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
== The &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS ==&lt;br /&gt;
&lt;br /&gt;
This QOS is specifically designed to be used by Mufasa users to &amp;#039;&amp;#039;&amp;#039;quickly build [[System#Containers|container images]]&amp;#039;&amp;#039;&amp;#039;. Its associated priority tier is very high: this, combined with the fact that jobs using this QOS require few resources, means that such jobs usually get executed very soon.&lt;br /&gt;
&lt;br /&gt;
On the other hand, the limited resources and the lack of access to the GPUs make the &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS unsuitable for other tasks.&lt;br /&gt;
&lt;br /&gt;
See [[Singularity#Building Singularity images|Building Singularity images]] for directions about building Singularity container images.&lt;br /&gt;
&lt;br /&gt;
== Restricted QOSes ==&lt;br /&gt;
&lt;br /&gt;
In Mufasa, the most powerful QOSes are reserved to researchers (called &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users: these include academic personnel and Ph.D. students). M.Sc. students, called &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users, cannot use them while running jobs.&lt;br /&gt;
&lt;br /&gt;
See [[#research users and students users|below]] to understand how &amp;#039;&amp;#039;user categories&amp;#039;&amp;#039; work in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
= &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users =&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa belong to two &amp;#039;&amp;#039;&amp;#039;user categories&amp;#039;&amp;#039;&amp;#039;. Categories provide users with different access to  to Mufasa&amp;#039;s resources: the idea being to provide researchers with more access while still letting students benefit from the server&amp;#039;s capabilities.&lt;br /&gt;
&lt;br /&gt;
User categories in Mufasa are the following:&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. academic personnel and Ph.D. students&lt;br /&gt;
::: * can use all QOSes, including the the [[#Restricted QOSes|restricted ones]]&lt;br /&gt;
::: * their jobs have a higher &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a higher number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. M.Sc. students&lt;br /&gt;
::: * cannot use the [[#Restricted QOSes|restricted QOSes]]&lt;br /&gt;
::: * their jobs have a lower &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a lower number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
You can inspect the differences in priority and running jobs between &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association format=&amp;quot;account,priority,maxjobs&amp;quot; | grep -E &amp;#039;Account|research|students&amp;#039;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
   Account   Priority MaxJobs &lt;br /&gt;
  research          4       2 &lt;br /&gt;
  students          1       1&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
To know what limits apply to your own user, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association where user=$USER format=&amp;quot;user,priority,maxjobs,qos%-60&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
      User   Priority MaxJobs QOS                                                          &lt;br /&gt;
---------- ---------- ------- ------------------------------------------------------------ &lt;br /&gt;
    preali          4       2 build,gpu,gpuheavy-20,gpuheavy-40,gpulight,gpuwide,nogpu&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The list under &amp;quot;QOS&amp;quot; shows what QOSes your user is allowed to use when [[User Jobs|running jobs]]. &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users can use all of them, while &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users can only access a subset of them.&lt;br /&gt;
&lt;br /&gt;
= Limits on jobs by the same user =&lt;br /&gt;
&lt;br /&gt;
Mufasa enforces limits on the number of jobs from a single user. Such limits aim at preventing users from &amp;quot;hogging&amp;quot; system resources, and apply to:&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;submitted jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that the user asked SLURM to execute, each of which may currently be either running or queued&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;running jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that are currently in execution&lt;br /&gt;
&lt;br /&gt;
Limits on submitted jobs are set via [[#SLURM Quality of Service (QOS)|QOSes]], while limits on running jobs are set via [[#research users and students users|user category]].&lt;br /&gt;
&lt;br /&gt;
The following table summarises what limits exist:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! &lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;running&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;submitted&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | global limits&amp;lt;br/&amp;gt;(system-wide)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to limits set by individual QOSes (below)&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | limits for&amp;lt;br/&amp;gt;individual&amp;lt;/br&amp;gt;QOSes&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to global limits (above)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for the&amp;#039;&amp;#039; &amp;lt;code&amp;gt;gpuwide&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;QOS&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for each of the other QOSes&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
= Job priority =&lt;br /&gt;
&lt;br /&gt;
Once the execution of a job has been requested, the job is not run immediately: it is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039; by SLURM, together with all the other jobs awaiting execution. The job on top of the queue at any time is the first to be put into execution as soon as the resources it requires are available. &lt;br /&gt;
&lt;br /&gt;
The order of the items in the job queue depends on their &amp;#039;&amp;#039;&amp;#039;priority&amp;#039;&amp;#039;&amp;#039;. The job with the highest priority is put by SLURM on top of the queue, while all other jobs are put in the queue in order of descending priority.&lt;br /&gt;
&lt;br /&gt;
The goal of SLURM is to maximise resource availability: i.e., to ensure the shorter possible wait time before job execution. To achieve this goal, in Mufasa SLURM is configured to &amp;#039;&amp;#039;&amp;#039;encourage users not to ask for resources or execution time that their job doesn&amp;#039;t need&amp;#039;&amp;#039;&amp;#039;. This is done via the priority mechanism: the more resources and/or time a job requests, the lower its priority will be; and the later it will be executed.&lt;br /&gt;
&lt;br /&gt;
Priority management in Mufasa is designed to set up a &amp;#039;&amp;#039;&amp;#039;virtuous cycle&amp;#039;&amp;#039;&amp;#039; where users, by carefully choosing what to ask for, obtain two results:&lt;br /&gt;
* they ensure that their job is executed as soon as possible;&lt;br /&gt;
* they leave as much as possible of Mufasa&amp;#039;s resources free for other users&amp;#039;s jobs.&lt;br /&gt;
&lt;br /&gt;
== Elements determining job priority ==&lt;br /&gt;
In Mufasa, the priority of a job is computed by SLURM according to the following elements:&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#research users and students users|User category]]&amp;#039;&amp;#039;&amp;#039; (i.e., &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;)&lt;br /&gt;
::: Used to provide higher priority to jobs run by &amp;#039;&amp;#039;&amp;#039;research personnel&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#SLURM Quality of Service (QOS)|QOS]]&amp;#039;&amp;#039;&amp;#039; used by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;less resources&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Number of CPUs&amp;#039;&amp;#039;&amp;#039; requested by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;a lower number of CPUs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job Age&amp;#039;&amp;#039;&amp;#039;, i.e. the time that the job has been waiting in the queue&lt;br /&gt;
::: Used to provide higher priority to jobs which have been &amp;#039;&amp;#039;&amp;#039;queued for a longer time&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
::: Note: periods when a job is not eligible for running because of the [[#Limits on jobs by the same user|limits on running jobs by the same user]] do not increase job age.&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;FairShare&amp;#039;&amp;#039;&amp;#039;, i.e. a factor computed by SLURM to balance use of the system by different users&lt;br /&gt;
::: Used to provide higher priority to jobs by users who &amp;#039;&amp;#039;&amp;#039;used Mufasa&amp;#039;s resources less than others&amp;#039;&amp;#039;&amp;#039; (see [[#How FairShare works|below]])&lt;br /&gt;
&lt;br /&gt;
You can understand how each of these elements contributes to the priority of the jobs waiting for execution (both yours and of other users) with command &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;. A tailored version of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; that only displays the parameters listed above is the following:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sprio -o &amp;quot;%6i %9u %.8Y %.10B %.5Q %.9T %.5A %.5F&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Here is an example of the output of the command above:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
JOBID  USER      PRIORITY      ASSOC   QOS      TRES   AGE FAIRS&lt;br /&gt;
5790   amoglia       3495       2500   375  cpu=-800   933   487&lt;br /&gt;
5791   qliu          3010       2500   375  cpu=-800   916    19&lt;br /&gt;
5817   mzoccola      2336        625   750  cpu=-800   495  1266&lt;br /&gt;
5822   fbottazzi     2235        625   750  cpu=-600   467   994&lt;br /&gt;
5837   lpassoni      2508       2500   750  cpu=-800     0    58&lt;br /&gt;
5838   bagostini     4437        625  3000  cpu=-200     0  1013&lt;br /&gt;
5839   flazzaro      1549        625   750  cpu=-800     0   974&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
In it, column PRIORITY provides the overall priority of each job. Each priority value is the sum of the five values to its right; in particular, column ASSOC corresponds to user category, while column TRES corresponds to the number of CPUs. Note that the contribution of TRES to priority is &amp;#039;&amp;#039;negative&amp;#039;&amp;#039;, i.e. it is a penalty; its value is proportional to the number of CPUs requested by the job.&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;Things to consider when using &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
* Its output is only relevant for jobs &amp;#039;&amp;#039;not yet in execution&amp;#039;&amp;#039;. For an already running job, the values shown by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; correspond to the priority that the job would get if evaluated now: however, this priority does not affect job execution in any way.&lt;br /&gt;
* Jobs listed by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; may temporarily differ from jobs listed by &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;: this is due to the fact that SLURM updates its internal data (used by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;) every few minutes, while the output of &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt; is updated much more frequently.&lt;br /&gt;
&lt;br /&gt;
=== How FairShare works ===&lt;br /&gt;
In Mufasa 2.0, the FairShare of each user is computed according to these rules:&lt;br /&gt;
# It considers the quantity used by (current and past) user jobs of:&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;CPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of CPUs used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;RAM&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the amount of RAM used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;GPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of GPUs used&lt;br /&gt;
#** note: 40 GB GPUs have double the impact on FairShare than 20 GB GPUs&lt;br /&gt;
# The impact on FairShare of any use of resources is &amp;#039;&amp;#039;&amp;#039;proportional to its duration&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
#* example: using 32 GB of RAM for 48h has the same impact of using 64 GB of RAM for 24h&lt;br /&gt;
# FairShare has a &amp;quot;fading memory&amp;quot;: i.e., resource use has less and less impact on FairShare the farther in the past it is&lt;br /&gt;
#* consequence: over time, the &amp;quot;history&amp;quot; of a user gets forgotten by FairShare&lt;br /&gt;
&lt;br /&gt;
== How to maximise the priority of your jobs ==&lt;br /&gt;
&lt;br /&gt;
Every time you run a SLURM job, follow these guidelines:&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Choose the less powerful QOS compatible with the needs of your job&lt;br /&gt;
:: QOSes with access to less resources have higher base priority&lt;br /&gt;
&lt;br /&gt;
; Only request CPUs and RAM that your job really needs&lt;br /&gt;
:: Usually, code exploits multiple CPUs only if specifically designed to do so: if unsure, only ask for 1 CPU&lt;br /&gt;
:: The fewer resources your job asks for, the less it will wait before they become available&lt;br /&gt;
:: The more CPUs your job asks for, the stronger the penalty to its priority&lt;br /&gt;
:: Asking for fewer CPUs and/or less RAM improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Do not request more time than your jobs needs to complete&lt;br /&gt;
:: Make a worst-case estimate and only ask for that duration&lt;br /&gt;
:: Asking for less of Mufasa&amp;#039;s time improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Debug and test code using less powerful QOSes before running it with more powerful QOSes&lt;br /&gt;
:: Your test jobs will get a higher priority and your FairShare will improve&lt;br /&gt;
&lt;br /&gt;
; Cancel jobs that you don&amp;#039;t need them anymore&lt;br /&gt;
:: Always use [[User_Jobs#Cancelling_a_job_with_scancel|scancel]] to delete completed (or crashed) jobs: your Fairshare will improve&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Protip: it&amp;#039;s a good idea to [[User Jobs#Looking for unused GPUs|check for unused GPUs]] before choosing what to request. Requesting a GPU that is currently idle will help your job get executed sooner.&lt;br /&gt;
&lt;br /&gt;
= SLURM partitions =&lt;br /&gt;
&lt;br /&gt;
Partitions are another mechanism provided by SLURM to create different levels of access to system resources. Since in Mufasa 2.0 access to resources is controlled via [[#SLURM Quality of Service (QOS)|QOSes]], partitions are not very relevant. (This is one of the main differences between Mufasa 1.0 and Mufasa 2.0.)&lt;br /&gt;
&lt;br /&gt;
Note, however, that the default values for some features of SLURM jobs (e.g., duration) are [[#Default values|set by the partition]].&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, there is a single SLURM partition, called &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt;, and all jobs run on it. The state of &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt; can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sinfo -o &amp;quot;%10P %5a %9T %11L %10l&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PARTITION  AVAIL STATE     DEFAULTTIME TIMELIMIT &lt;br /&gt;
jobs*      up    idle      1:00:00     3-00:00:00&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where columns correspond to the following information:&lt;br /&gt;
&lt;br /&gt;
:; PARTITION&lt;br /&gt;
:: name of the partition; the asterisks indicates that it&amp;#039;s the default one&lt;br /&gt;
&lt;br /&gt;
:; AVAIL&lt;br /&gt;
:: state/availability of the partition: see [[#Partition availability|below]]&lt;br /&gt;
&lt;br /&gt;
:; STATE&lt;br /&gt;
:: state (using [https://slurm.schedmd.com/sinfo.html#SECTION_NODE-STATE-CODES these codes])&lt;br /&gt;
:: typical values are &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;mixed&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that some of the resources are busy executing jobs while other are idle, and &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;allocated&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that all of the resources are in use&lt;br /&gt;
&lt;br /&gt;
:; DEFAULTTIME&lt;br /&gt;
:: default runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:; TIMELIMIT&lt;br /&gt;
:: maximum runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
The asterisk at the end of the partition name indicates the default partition, i.e. the one on which jobs which do not ask for a specific partition are run.&lt;br /&gt;
&lt;br /&gt;
Command &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; does not tell you about the &amp;#039;&amp;#039;jobs&amp;#039;&amp;#039; submitted to a partition. This information is obtained, instead, with [[User Jobs#Inspecting jobs with squeue|command &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;]].&lt;br /&gt;
&lt;br /&gt;
== Partition availability ==&lt;br /&gt;
&lt;br /&gt;
The most important information that &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; provides is the &amp;#039;&amp;#039;&amp;#039;availability&amp;#039;&amp;#039;&amp;#039; (also called &amp;#039;&amp;#039;&amp;#039;state&amp;#039;&amp;#039;&amp;#039;) of partitions. This is shown in column &amp;quot;AVAIL&amp;quot;. Possible partition states are:&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is available&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed as soon as resources allow&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is in the process of becoming unavailable (i.e., of entering the &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; state: see below)&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is unavailable&lt;br /&gt;
:: There are no running jobs&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
When a partition goes from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; no harm is done to running jobs. In a normally functioning SLURM system, the passage from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; happens only when no jobs are running on the partition. If (e.g., due to a malfunction) the passage happens with jobs still running, they get killed.&lt;br /&gt;
&lt;br /&gt;
A partition in state &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; requires intervention by a [[Roles|Job Administrator]] to be restored to &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
== Default values ==&lt;br /&gt;
&lt;br /&gt;
The features of SLURM partitions, including the &amp;#039;&amp;#039;&amp;#039;default values&amp;#039;&amp;#039;&amp;#039; which are applied to jobs that do not make explicit requests, can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scontrol show partition&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PartitionName=jobs&lt;br /&gt;
   AllowGroups=ALL AllowAccounts=ALL AllowQos=nogpu,gpulight,gpu,gpuwide,gpuheavy-20,gpuheavy-40&lt;br /&gt;
   AllocNodes=ALL Default=YES QoS=N/A&lt;br /&gt;
   DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO&lt;br /&gt;
   MaxNodes=UNLIMITED MaxTime=3-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED&lt;br /&gt;
   Nodes=gn01&lt;br /&gt;
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO&lt;br /&gt;
   OverTimeLimit=NONE PreemptMode=OFF&lt;br /&gt;
   State=UP TotalCPUs=48 TotalNodes=1 SelectTypeParameters=NONE&lt;br /&gt;
   JobDefaults=(null)&lt;br /&gt;
   DefMemPerNode=4096 MaxMemPerNode=UNLIMITED&lt;br /&gt;
   TRES=cpu=48,mem=1011435M,node=1,billing=49,gres/gpu=13,gres/gpu:3g.20gb=5,gres/gpu:40gb=3,gres/gpu:4g.20gb=5&lt;br /&gt;
   TRESBillingWeights=cpu=1.0,gres/gpu:3g.20gb=6.0,gres/gpu:4g.20gb=6.0,gres/gpu:40gb=6.0,mem=0.05g&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In the output of &amp;lt;code&amp;gt;scontrol show partition&amp;lt;/code&amp;gt; the most relevant default values for Mufasa users are:&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefaultTime&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default execution time assigned to a job run on the partition (e.g., 1 hour)&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefMemPerNode&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default amount of RAM assigned to a job run on the partition (e.g., 4GB)&lt;br /&gt;
&lt;br /&gt;
= System resources subjected to limitations =&lt;br /&gt;
&lt;br /&gt;
In systems based on SLURM like Mufasa, &amp;#039;&amp;#039;&amp;#039;TRES (Trackable RESources)&amp;#039;&amp;#039;&amp;#039; are (from [https://slurm.schedmd.com/tres.html SLURM&amp;#039;s documentation] &amp;quot;&amp;#039;&amp;#039;resources that can be tracked for usage or used to enforce limits against.&amp;#039;&amp;#039;&amp;quot;&lt;br /&gt;
&lt;br /&gt;
TRES include CPUs, RAM and &amp;#039;&amp;#039;&amp;#039;GRES&amp;#039;&amp;#039;&amp;#039;. The last term stands for &amp;#039;&amp;#039;Generic RESources&amp;#039;&amp;#039; that a job may need for its execution. In Mufasa, the only &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; resources are the GPUs.&lt;br /&gt;
&lt;br /&gt;
== &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax ==&lt;br /&gt;
&lt;br /&gt;
To ask SLURM to assign GRES resources (i.e., GPUs) to a job, a special syntax must be used. Precisely, the name of each GPU resource takes the form&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:Name:Type&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
Considering the [[System#CPUs and GPUs|GPU complement of Mufasa]], &amp;lt;code&amp;gt;Type&amp;lt;/code&amp;gt; takes the following values:&lt;br /&gt;
&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:40gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 40 Gbytes of RAM&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:4g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 4 compute units&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 3 compute units&lt;br /&gt;
&lt;br /&gt;
So, for instance,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
identifies a resource corresponding to a GPU with 20 GB of RAM and 3 compute units.&lt;br /&gt;
&lt;br /&gt;
When asking for a GRES resource (e.g., in an &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command or an &amp;lt;code&amp;gt;SBATCH&amp;lt;/code&amp;gt; directive of an [[User Jobs#Using execution scripts to run jobs|execution script]]), the syntax required by SLURM is&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:&amp;lt;Type&amp;gt;:&amp;lt;Quantity&amp;gt;&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
where &amp;lt;code&amp;gt;Quantity&amp;lt;/code&amp;gt; is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of type &amp;lt;code&amp;gt;4g.20gb&amp;lt;/code&amp;gt; the syntax is&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:4g.20gb:2&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SLURM&amp;#039;s &amp;#039;&amp;#039;generic resources&amp;#039;&amp;#039; are defined in &amp;lt;code&amp;gt;/etc/slurm/gres.conf&amp;lt;/code&amp;gt;. In order to make GPUs available to SLURM&amp;#039;s &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; management, Mufasa makes use of Nvidia&amp;#039;s [https://developer.nvidia.com/nvidia-management-library-nvml NVML library]. For additional information see [https://slurm.schedmd.com/gres.html SLURM&amp;#039;s documentation].&lt;/div&gt;</summary>
		<author><name>GiulioFontana</name></author>
	</entry>
	<entry>
		<id>https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2597</id>
		<title>SLURM</title>
		<link rel="alternate" type="text/html" href="https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2597"/>
		<updated>2026-11-18T09:05:54Z</updated>

		<summary type="html">&lt;p&gt;GiulioFontana: /* Elements determining job priority */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page presents the features of SLURM that are most relevant to Mufasa&amp;#039;s [[Roles|Job Users]]. Job Users can submit jobs for execution, cancel their own jobs, and see other users&amp;#039; jobs (but not intervene on them).&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa &amp;#039;&amp;#039;&amp;#039;must use SLURM&amp;#039;&amp;#039;&amp;#039; to run resource-heavy processes, i.e. computing jobs that require one or more of the following:&lt;br /&gt;
* GPUs&lt;br /&gt;
* multiple CPUs&lt;br /&gt;
* powerful CPUs&lt;br /&gt;
* a significant amount of RAM&lt;br /&gt;
&lt;br /&gt;
In fact, only processes run via SLURM have access to all the resources of Mufasa. Processes run outside SLURM are executed by the [[System#Login server|login server]] virtual machine, which has minimal resources and no GPUs. Using SLURM is therefore the only way to execute resource-heavy jobs on Mufasa (this is a key difference between Mufasa 1.0 and Mufasa 2.0).&lt;br /&gt;
&lt;br /&gt;
= SLURM in a nutshell =&lt;br /&gt;
&lt;br /&gt;
Computation jobs on Mufasa needs to be launched via [[System#The SLURM job scheduling system|SLURM]]. SLURM provides jobs with access to the [[#System resources subjected to limitations|physical resources]] of Mufasa, such as CPUs, GPUs and RAM. Thanks to SLURM, processing jobs share system resources, optimising their occupation and availability. &lt;br /&gt;
&lt;br /&gt;
When a user runs a job, the job does not get executed immediately and is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039;. SLURM executes jobs according to their order in the queue: the top job in the queue gets executed as soon as the necessary resources are available, while jobs lower in the queue wait longer. The position of a job in the queue is due to the &amp;#039;&amp;#039;&amp;#039;[[#Job priority|priority]]&amp;#039;&amp;#039;&amp;#039; assigned to it by SLURM, with higher-priority jobs closer to the top. As a general rule:&lt;br /&gt;
&lt;br /&gt;
;: &amp;#039;&amp;#039;&amp;#039;the greater the fraction of Mufasa&amp;#039;s overall resources that a job asks for, the lower the job&amp;#039;s priority will be&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
The priority mechanism is used to encourage users to use Mufasa&amp;#039;s resources (i.e.: GPUs, CPUs, RAM, execution time) in an effective and equitable manner. This page includes a [[#How_to_maximise_the_priority_of_your_jobs|chart explaining how to maximise the priority of your jobs]].&lt;br /&gt;
&lt;br /&gt;
The &amp;#039;&amp;#039;&amp;#039;time&amp;#039;&amp;#039;&amp;#039; available to a job for its execution is controlled by SLURM. When a user requests execution of a job, they must specify the duration of the time slot that the job needs. The job must complete its execution before the end of the requested time slot, because at the end of such slot it gets killed by SLURM.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0 access to system resources is managed via SLURM&amp;#039;s [[#SLURM Quality of Service (QOS)|Quality of Service (QOS)]] mechanism (Mufasa 1.0 used [[#SLURM_partitions|partitions]] instead). While [[User Jobs#Running jobs with SLURM|launching a processing job via SLURM]], the user must always specify the chosen QOS. QOSes differ in the set of resources that they provide access to because each of them is designed to fit a given type of job.&lt;br /&gt;
&lt;br /&gt;
Mufasa sets limits to the number of jobs by the same user. This page includes a [[#Limits on jobs by the same user|table summarising such limits]].&lt;br /&gt;
&lt;br /&gt;
= SLURM Quality of Service (QOS) =&lt;br /&gt;
&lt;br /&gt;
Through &amp;#039;&amp;#039;&amp;#039;Quality of Services&amp;#039;&amp;#039;&amp;#039; (&amp;#039;&amp;#039;&amp;#039;QOSes&amp;#039;&amp;#039;&amp;#039;), SLURM lets system configurators assign a name to a set of related constraints.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, QOSes are used to define different levels of access to the server&amp;#039;s resources. When [[User Jobs|executing a job with SLURM]], a user must always &amp;#039;&amp;#039;&amp;#039;specify the QOS&amp;#039;&amp;#039;&amp;#039; that their job will use: this choice, in turn, determines what resources the job is able to access and influences the [[#Job priority|priority]] of the job.&lt;br /&gt;
&lt;br /&gt;
Mufasa&amp;#039;s QOSes and their features can be inspected with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list qos format=name%-11,priority,MaxSubmit,maxwall,maxtres%-80,mintres%-18&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following (see below for the same data in table form):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
Name          Priority MaxSubmit     MaxWall MaxTRES                                                                          MinTRES            &lt;br /&gt;
----------- ---------- --------- ----------- -------------------------------------------------------------------------------- ------------------ &lt;br /&gt;
normal               0                                                                                                                           &lt;br /&gt;
nogpu                4         1  3-00:00:00 cpu=16,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=128G                    &lt;br /&gt;
gpuheavy-20          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=2,mem=128G             gres/gpu:4g.20gb=1 &lt;br /&gt;
gpuheavy-40          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=1,gres/gpu:4g.20gb=0,mem=128G             gres/gpu:40gb=1    &lt;br /&gt;
gpulight             8         1    12:00:00 cpu=2,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpu                  2         1  1-00:00:00 cpu=8,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpuwide              2         2  1-00:00:00 cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=1,mem=64G              gres/gpu:4g.20gb=1 &lt;br /&gt;
build               32         1    02:00:00 cpu=2,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=16G&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The columns of this output are the following:&lt;br /&gt;
&lt;br /&gt;
:; Name&lt;br /&gt;
:: name of the QOS&lt;br /&gt;
&lt;br /&gt;
:; Priority&lt;br /&gt;
:: priority tier associated to the QOS (higher value = higher priority): see [[#Job priority|Job priority]] for details&lt;br /&gt;
&lt;br /&gt;
:; MaxSubmit&lt;br /&gt;
:: maximum number of jobs from a single user that can be submitted to SLURM with this QOS; submitted jobs include both running and queued jobs&lt;br /&gt;
:: See [[#Limits on jobs by the same user|Limits on jobs by the same user]] for an overview of the limits on jobs set by Mufasa.&lt;br /&gt;
&lt;br /&gt;
:; MaxWall&lt;br /&gt;
:: maximum wall clock duration of the jobs using the QOS (after which they are killed by SLURM), in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
:: For some QOSes these are not set: it means that they are determined by the [[#SLURM partitions|partition]]. Partitions also define the [[#Default values|default duration]] of jobs.&lt;br /&gt;
&lt;br /&gt;
:; MaxTRES&lt;br /&gt;
:: amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) available to a job using the QOS, where&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;cpu=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of CPUs (i.e., processor cores) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of CPUs specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;gres/&amp;#039;&amp;#039;gpu:Type&amp;#039;&amp;#039;=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of GPUs of class &amp;lt;code&amp;gt;&amp;#039;&amp;#039;Type&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; (see [[User Jobs#gres syntax|&amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax]]) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; (for QOSes that allow access to GPUs) if not specified, the job cannot be launched&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;mem=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;G&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum amount of system RAM is &amp;#039;&amp;#039;K&amp;#039;&amp;#039; GBytes&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of RAM specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
&lt;br /&gt;
:; MinTRES&lt;br /&gt;
:: minimum amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) that a job using the QOS must request in order to actually get executed by SLURM.&lt;br /&gt;
:: (If your job does not actually need these resources, you&amp;#039;ve chosen the wrong QOS: you can use one with a higher priority.)&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; QOS is the default one, and exists only to ensure that users always specify a QOS when running a job. Since &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; has zero priority and no resources, a job run using this QOS would never be run.&lt;br /&gt;
&lt;br /&gt;
The information provided by the &amp;lt;code&amp;gt;sacctmgr list qos&amp;lt;/code&amp;gt; command above is summarised by the following table:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! QOS&lt;br /&gt;
! Priority&amp;lt;/br&amp;gt;tier&lt;br /&gt;
! Max&amp;lt;/br&amp;gt;Submit&lt;br /&gt;
! MaxWall&amp;lt;/br&amp;gt;[h]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;CPUs&lt;br /&gt;
! max RAM&amp;lt;/br&amp;gt;[GB]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;3g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;4g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;40GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | build&lt;br /&gt;
| 32&lt;br /&gt;
| 1&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 16&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpulight&lt;br /&gt;
| 8&lt;br /&gt;
| 1&lt;br /&gt;
| 12&lt;br /&gt;
| 2&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | nogpu&lt;br /&gt;
| 4&lt;br /&gt;
| 1&lt;br /&gt;
| 72&lt;br /&gt;
| 16&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpu&lt;br /&gt;
| 2&lt;br /&gt;
| 1&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuwide&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-20&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| 2 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-40&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
A key piece of information in the table above is the &amp;#039;&amp;#039;&amp;#039;priority tier&amp;#039;&amp;#039;&amp;#039; associated to each QOS. In Mufasa 2.0, priority tiers are used to encourage users to use the &amp;#039;&amp;#039;&amp;#039;least powerful QOS that is compatible with their needs&amp;#039;&amp;#039;&amp;#039;, where &amp;quot;powerful&amp;quot; means &amp;quot;rich with resources&amp;quot;. Less powerful QOSes increase the priority of the jobs that use them, so these jobs tend to be executed sooner. See [[#Job priority|Job priority]] for details about how priority affects the execution order of jobs in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
== The &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS ==&lt;br /&gt;
&lt;br /&gt;
This QOS is specifically designed to be used by Mufasa users to &amp;#039;&amp;#039;&amp;#039;quickly build [[System#Containers|container images]]&amp;#039;&amp;#039;&amp;#039;. Its associated priority tier is very high: this, combined with the fact that jobs using this QOS require few resources, means that such jobs usually get executed very soon.&lt;br /&gt;
&lt;br /&gt;
On the other hand, the limited resources and the lack of access to the GPUs make the &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS unsuitable for other tasks.&lt;br /&gt;
&lt;br /&gt;
See [[Singularity#Building Singularity images|Building Singularity images]] for directions about building Singularity container images.&lt;br /&gt;
&lt;br /&gt;
== Restricted QOSes ==&lt;br /&gt;
&lt;br /&gt;
In Mufasa, the most powerful QOSes are reserved to researchers (called &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users: these include academic personnel and Ph.D. students). M.Sc. students, called &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users, cannot use them while running jobs.&lt;br /&gt;
&lt;br /&gt;
See [[#research users and students users|below]] to understand how &amp;#039;&amp;#039;user categories&amp;#039;&amp;#039; work in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
= &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users =&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa belong to two &amp;#039;&amp;#039;&amp;#039;user categories&amp;#039;&amp;#039;&amp;#039;. Categories provide users with different access to  to Mufasa&amp;#039;s resources: the idea being to provide researchers with more access while still letting students benefit from the server&amp;#039;s capabilities.&lt;br /&gt;
&lt;br /&gt;
User categories in Mufasa are the following:&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. academic personnel and Ph.D. students&lt;br /&gt;
::: * can use all QOSes, including the the [[#Restricted QOSes|restricted ones]]&lt;br /&gt;
::: * their jobs have a higher &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a higher number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. M.Sc. students&lt;br /&gt;
::: * cannot use the [[#Restricted QOSes|restricted QOSes]]&lt;br /&gt;
::: * their jobs have a lower &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a lower number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
You can inspect the differences in priority and running jobs between &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association format=&amp;quot;account,priority,maxjobs&amp;quot; | grep -E &amp;#039;Account|research|students&amp;#039;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
   Account   Priority MaxJobs &lt;br /&gt;
  research          4       2 &lt;br /&gt;
  students          1       1&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
To know what limits apply to your own user, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association where user=$USER format=&amp;quot;user,priority,maxjobs,qos%-60&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
      User   Priority MaxJobs QOS                                                          &lt;br /&gt;
---------- ---------- ------- ------------------------------------------------------------ &lt;br /&gt;
    preali          4       2 build,gpu,gpuheavy-20,gpuheavy-40,gpulight,gpuwide,nogpu&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The list under &amp;quot;QOS&amp;quot; shows what QOSes your user is allowed to use when [[User Jobs|running jobs]]. &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users can use all of them, while &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users can only access a subset of them.&lt;br /&gt;
&lt;br /&gt;
= Limits on jobs by the same user =&lt;br /&gt;
&lt;br /&gt;
Mufasa enforces limits on the number of jobs from a single user. Such limits aim at preventing users from &amp;quot;hogging&amp;quot; system resources, and apply to:&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;submitted jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that the user asked SLURM to execute, each of which may currently be either running or queued&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;running jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that are currently in execution&lt;br /&gt;
&lt;br /&gt;
Limits on submitted jobs are set via [[#SLURM Quality of Service (QOS)|QOSes]], while limits on running jobs are set via [[#research users and students users|user category]].&lt;br /&gt;
&lt;br /&gt;
The following table summarises what limits exist:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! &lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;running&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;submitted&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | global limits&amp;lt;br/&amp;gt;(system-wide)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to limits set by individual QOSes (below)&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | limits for&amp;lt;br/&amp;gt;individual&amp;lt;/br&amp;gt;QOSes&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to global limits (above)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for the&amp;#039;&amp;#039; &amp;lt;code&amp;gt;gpuwide&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;QOS&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for each of the other QOSes&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
= Job priority =&lt;br /&gt;
&lt;br /&gt;
Once the execution of a job has been requested, the job is not run immediately: it is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039; by SLURM, together with all the other jobs awaiting execution. The job on top of the queue at any time is the first to be put into execution as soon as the resources it requires are available. &lt;br /&gt;
&lt;br /&gt;
The order of the items in the job queue depends on their &amp;#039;&amp;#039;&amp;#039;priority&amp;#039;&amp;#039;&amp;#039;. The job with the highest priority is put by SLURM on top of the queue, while all other jobs are put in the queue in order of descending priority.&lt;br /&gt;
&lt;br /&gt;
The goal of SLURM is to maximise resource availability: i.e., to ensure the shorter possible wait time before job execution. To achieve this goal, in Mufasa SLURM is configured to &amp;#039;&amp;#039;&amp;#039;encourage users not to ask for resources or execution time that their job doesn&amp;#039;t need&amp;#039;&amp;#039;&amp;#039;. This is done via the priority mechanism: the more resources and/or time a job requests, the lower its priority will be; and the later it will be executed.&lt;br /&gt;
&lt;br /&gt;
Priority management in Mufasa is designed to set up a &amp;#039;&amp;#039;&amp;#039;virtuous cycle&amp;#039;&amp;#039;&amp;#039; where users, by carefully choosing what to ask for, obtain two results:&lt;br /&gt;
* they ensure that their job is executed as soon as possible;&lt;br /&gt;
* they leave as much as possible of Mufasa&amp;#039;s resources free for other users&amp;#039;s jobs.&lt;br /&gt;
&lt;br /&gt;
== Elements determining job priority ==&lt;br /&gt;
In Mufasa, the priority of a job is computed by SLURM according to the following elements:&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#research users and students users|User category]]&amp;#039;&amp;#039;&amp;#039; (i.e., &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;)&lt;br /&gt;
::: Used to provide higher priority to jobs run by &amp;#039;&amp;#039;&amp;#039;research personnel&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#SLURM Quality of Service (QOS)|QOS]]&amp;#039;&amp;#039;&amp;#039; used by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;less resources&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Number of CPUs&amp;#039;&amp;#039;&amp;#039; requested by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;a lower number of CPUs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job Age&amp;#039;&amp;#039;&amp;#039;, i.e. the time that the job has been waiting in the queue&lt;br /&gt;
::: Used to provide higher priority to jobs which have been &amp;#039;&amp;#039;&amp;#039;queued for a longer time&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
::: Note: periods when a job is not eligible for running because of the [[#Limits on jobs by the same user|limits on running jobs by the same user]] do not increase job age.&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;FairShare&amp;#039;&amp;#039;&amp;#039;, i.e. a factor computed by SLURM to balance use of the system by different users&lt;br /&gt;
::: Used to provide higher priority to jobs by users who &amp;#039;&amp;#039;&amp;#039;used Mufasa&amp;#039;s resources less than others&amp;#039;&amp;#039;&amp;#039; (see [[#How FairShare works|below]])&lt;br /&gt;
&lt;br /&gt;
You can understand how each of these elements contributes to the priority of the jobs waiting for execution (both yours and of other users) with command &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;. A tailored version of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; that only displays the parameters listed above is the following:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sprio -o &amp;quot;%6i %9u %.8Y %.10B %.5Q %.9T %.5A %.5F&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Here is an example of the output of the command above:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
JOBID  USER      PRIORITY      ASSOC   QOS      TRES   AGE FAIRS&lt;br /&gt;
5790   amoglia       3495       2500   375  cpu=-800   933   487&lt;br /&gt;
5791   qliu          3010       2500   375  cpu=-800   916    19&lt;br /&gt;
5817   mzoccola      2336        625   750  cpu=-800   495  1266&lt;br /&gt;
5822   fbottazzi     2235        625   750  cpu=-600   467   994&lt;br /&gt;
5837   lpassoni      2508       2500   750  cpu=-800     0    58&lt;br /&gt;
5838   bagostini     4437        625  3000  cpu=-200     0  1013&lt;br /&gt;
5839   flazzaro      1549        625   750  cpu=-800     0   974&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
In it, column PRIORITY provides the overall priority of each job. Each priority value is the sum of the five values to its right; in particular, column ASSOC corresponds to user category, while column TRES corresponds to the number of CPUs. Note that the contribution of TRES to priority is &amp;#039;&amp;#039;negative&amp;#039;&amp;#039;, i.e. it is a penalty; its value is proportional to the number of CPUs requested by the job.&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;Things to consider when using &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
* Its output is only relevant for jobs &amp;#039;&amp;#039;not yet in execution&amp;#039;&amp;#039;. For an already running job, the values shown by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; correspond to the priority that the job would get if evaluated now, which generally differs from its actual priority.&lt;br /&gt;
* Jobs listed by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; may temporarily differ from jobs listed by &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;: this is due to the fact that SLURM updates its internal data with a periodicity of a few minutes.&lt;br /&gt;
&lt;br /&gt;
=== How FairShare works ===&lt;br /&gt;
In Mufasa 2.0, the FairShare of each user is computed according to these rules:&lt;br /&gt;
# It considers the quantity used by (current and past) user jobs of:&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;CPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of CPUs used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;RAM&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the amount of RAM used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;GPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of GPUs used&lt;br /&gt;
#** note: 40 GB GPUs have double the impact on FairShare than 20 GB GPUs&lt;br /&gt;
# The impact on FairShare of any use of resources is &amp;#039;&amp;#039;&amp;#039;proportional to its duration&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
#* example: using 32 GB of RAM for 48h has the same impact of using 64 GB of RAM for 24h&lt;br /&gt;
# FairShare has a &amp;quot;fading memory&amp;quot;: i.e., resource use has less and less impact on FairShare the farther in the past it is&lt;br /&gt;
#* consequence: over time, the &amp;quot;history&amp;quot; of a user gets forgotten by FairShare&lt;br /&gt;
&lt;br /&gt;
== How to maximise the priority of your jobs ==&lt;br /&gt;
&lt;br /&gt;
Every time you run a SLURM job, follow these guidelines:&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Choose the less powerful QOS compatible with the needs of your job&lt;br /&gt;
:: QOSes with access to less resources have higher base priority&lt;br /&gt;
&lt;br /&gt;
; Only request CPUs and RAM that your job really needs&lt;br /&gt;
:: Usually, code exploits multiple CPUs only if specifically designed to do so: if unsure, only ask for 1 CPU&lt;br /&gt;
:: The fewer resources your job asks for, the less it will wait before they become available&lt;br /&gt;
:: The more CPUs your job asks for, the stronger the penalty to its priority&lt;br /&gt;
:: Asking for fewer CPUs and/or less RAM improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Do not request more time than your jobs needs to complete&lt;br /&gt;
:: Make a worst-case estimate and only ask for that duration&lt;br /&gt;
:: Asking for less of Mufasa&amp;#039;s time improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Debug and test code using less powerful QOSes before running it with more powerful QOSes&lt;br /&gt;
:: Your test jobs will get a higher priority and your FairShare will improve&lt;br /&gt;
&lt;br /&gt;
; Cancel jobs that you don&amp;#039;t need them anymore&lt;br /&gt;
:: Always use [[User_Jobs#Cancelling_a_job_with_scancel|scancel]] to delete completed (or crashed) jobs: your Fairshare will improve&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Protip: it&amp;#039;s a good idea to [[User Jobs#Looking for unused GPUs|check for unused GPUs]] before choosing what to request. Requesting a GPU that is currently idle will help your job get executed sooner.&lt;br /&gt;
&lt;br /&gt;
= SLURM partitions =&lt;br /&gt;
&lt;br /&gt;
Partitions are another mechanism provided by SLURM to create different levels of access to system resources. Since in Mufasa 2.0 access to resources is controlled via [[#SLURM Quality of Service (QOS)|QOSes]], partitions are not very relevant. (This is one of the main differences between Mufasa 1.0 and Mufasa 2.0.)&lt;br /&gt;
&lt;br /&gt;
Note, however, that the default values for some features of SLURM jobs (e.g., duration) are [[#Default values|set by the partition]].&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, there is a single SLURM partition, called &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt;, and all jobs run on it. The state of &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt; can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sinfo -o &amp;quot;%10P %5a %9T %11L %10l&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PARTITION  AVAIL STATE     DEFAULTTIME TIMELIMIT &lt;br /&gt;
jobs*      up    idle      1:00:00     3-00:00:00&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where columns correspond to the following information:&lt;br /&gt;
&lt;br /&gt;
:; PARTITION&lt;br /&gt;
:: name of the partition; the asterisks indicates that it&amp;#039;s the default one&lt;br /&gt;
&lt;br /&gt;
:; AVAIL&lt;br /&gt;
:: state/availability of the partition: see [[#Partition availability|below]]&lt;br /&gt;
&lt;br /&gt;
:; STATE&lt;br /&gt;
:: state (using [https://slurm.schedmd.com/sinfo.html#SECTION_NODE-STATE-CODES these codes])&lt;br /&gt;
:: typical values are &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;mixed&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that some of the resources are busy executing jobs while other are idle, and &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;allocated&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that all of the resources are in use&lt;br /&gt;
&lt;br /&gt;
:; DEFAULTTIME&lt;br /&gt;
:: default runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:; TIMELIMIT&lt;br /&gt;
:: maximum runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
The asterisk at the end of the partition name indicates the default partition, i.e. the one on which jobs which do not ask for a specific partition are run.&lt;br /&gt;
&lt;br /&gt;
Command &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; does not tell you about the &amp;#039;&amp;#039;jobs&amp;#039;&amp;#039; submitted to a partition. This information is obtained, instead, with [[User Jobs#Inspecting jobs with squeue|command &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;]].&lt;br /&gt;
&lt;br /&gt;
== Partition availability ==&lt;br /&gt;
&lt;br /&gt;
The most important information that &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; provides is the &amp;#039;&amp;#039;&amp;#039;availability&amp;#039;&amp;#039;&amp;#039; (also called &amp;#039;&amp;#039;&amp;#039;state&amp;#039;&amp;#039;&amp;#039;) of partitions. This is shown in column &amp;quot;AVAIL&amp;quot;. Possible partition states are:&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is available&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed as soon as resources allow&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is in the process of becoming unavailable (i.e., of entering the &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; state: see below)&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is unavailable&lt;br /&gt;
:: There are no running jobs&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
When a partition goes from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; no harm is done to running jobs. In a normally functioning SLURM system, the passage from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; happens only when no jobs are running on the partition. If (e.g., due to a malfunction) the passage happens with jobs still running, they get killed.&lt;br /&gt;
&lt;br /&gt;
A partition in state &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; requires intervention by a [[Roles|Job Administrator]] to be restored to &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
== Default values ==&lt;br /&gt;
&lt;br /&gt;
The features of SLURM partitions, including the &amp;#039;&amp;#039;&amp;#039;default values&amp;#039;&amp;#039;&amp;#039; which are applied to jobs that do not make explicit requests, can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scontrol show partition&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PartitionName=jobs&lt;br /&gt;
   AllowGroups=ALL AllowAccounts=ALL AllowQos=nogpu,gpulight,gpu,gpuwide,gpuheavy-20,gpuheavy-40&lt;br /&gt;
   AllocNodes=ALL Default=YES QoS=N/A&lt;br /&gt;
   DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO&lt;br /&gt;
   MaxNodes=UNLIMITED MaxTime=3-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED&lt;br /&gt;
   Nodes=gn01&lt;br /&gt;
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO&lt;br /&gt;
   OverTimeLimit=NONE PreemptMode=OFF&lt;br /&gt;
   State=UP TotalCPUs=48 TotalNodes=1 SelectTypeParameters=NONE&lt;br /&gt;
   JobDefaults=(null)&lt;br /&gt;
   DefMemPerNode=4096 MaxMemPerNode=UNLIMITED&lt;br /&gt;
   TRES=cpu=48,mem=1011435M,node=1,billing=49,gres/gpu=13,gres/gpu:3g.20gb=5,gres/gpu:40gb=3,gres/gpu:4g.20gb=5&lt;br /&gt;
   TRESBillingWeights=cpu=1.0,gres/gpu:3g.20gb=6.0,gres/gpu:4g.20gb=6.0,gres/gpu:40gb=6.0,mem=0.05g&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In the output of &amp;lt;code&amp;gt;scontrol show partition&amp;lt;/code&amp;gt; the most relevant default values for Mufasa users are:&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefaultTime&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default execution time assigned to a job run on the partition (e.g., 1 hour)&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefMemPerNode&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default amount of RAM assigned to a job run on the partition (e.g., 4GB)&lt;br /&gt;
&lt;br /&gt;
= System resources subjected to limitations =&lt;br /&gt;
&lt;br /&gt;
In systems based on SLURM like Mufasa, &amp;#039;&amp;#039;&amp;#039;TRES (Trackable RESources)&amp;#039;&amp;#039;&amp;#039; are (from [https://slurm.schedmd.com/tres.html SLURM&amp;#039;s documentation] &amp;quot;&amp;#039;&amp;#039;resources that can be tracked for usage or used to enforce limits against.&amp;#039;&amp;#039;&amp;quot;&lt;br /&gt;
&lt;br /&gt;
TRES include CPUs, RAM and &amp;#039;&amp;#039;&amp;#039;GRES&amp;#039;&amp;#039;&amp;#039;. The last term stands for &amp;#039;&amp;#039;Generic RESources&amp;#039;&amp;#039; that a job may need for its execution. In Mufasa, the only &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; resources are the GPUs.&lt;br /&gt;
&lt;br /&gt;
== &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax ==&lt;br /&gt;
&lt;br /&gt;
To ask SLURM to assign GRES resources (i.e., GPUs) to a job, a special syntax must be used. Precisely, the name of each GPU resource takes the form&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:Name:Type&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
Considering the [[System#CPUs and GPUs|GPU complement of Mufasa]], &amp;lt;code&amp;gt;Type&amp;lt;/code&amp;gt; takes the following values:&lt;br /&gt;
&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:40gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 40 Gbytes of RAM&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:4g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 4 compute units&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 3 compute units&lt;br /&gt;
&lt;br /&gt;
So, for instance,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
identifies a resource corresponding to a GPU with 20 GB of RAM and 3 compute units.&lt;br /&gt;
&lt;br /&gt;
When asking for a GRES resource (e.g., in an &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command or an &amp;lt;code&amp;gt;SBATCH&amp;lt;/code&amp;gt; directive of an [[User Jobs#Using execution scripts to run jobs|execution script]]), the syntax required by SLURM is&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:&amp;lt;Type&amp;gt;:&amp;lt;Quantity&amp;gt;&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
where &amp;lt;code&amp;gt;Quantity&amp;lt;/code&amp;gt; is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of type &amp;lt;code&amp;gt;4g.20gb&amp;lt;/code&amp;gt; the syntax is&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:4g.20gb:2&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SLURM&amp;#039;s &amp;#039;&amp;#039;generic resources&amp;#039;&amp;#039; are defined in &amp;lt;code&amp;gt;/etc/slurm/gres.conf&amp;lt;/code&amp;gt;. In order to make GPUs available to SLURM&amp;#039;s &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; management, Mufasa makes use of Nvidia&amp;#039;s [https://developer.nvidia.com/nvidia-management-library-nvml NVML library]. For additional information see [https://slurm.schedmd.com/gres.html SLURM&amp;#039;s documentation].&lt;/div&gt;</summary>
		<author><name>GiulioFontana</name></author>
	</entry>
	<entry>
		<id>https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2596</id>
		<title>SLURM</title>
		<link rel="alternate" type="text/html" href="https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2596"/>
		<updated>2026-11-18T09:04:37Z</updated>

		<summary type="html">&lt;p&gt;GiulioFontana: /* Elements determining job priority */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page presents the features of SLURM that are most relevant to Mufasa&amp;#039;s [[Roles|Job Users]]. Job Users can submit jobs for execution, cancel their own jobs, and see other users&amp;#039; jobs (but not intervene on them).&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa &amp;#039;&amp;#039;&amp;#039;must use SLURM&amp;#039;&amp;#039;&amp;#039; to run resource-heavy processes, i.e. computing jobs that require one or more of the following:&lt;br /&gt;
* GPUs&lt;br /&gt;
* multiple CPUs&lt;br /&gt;
* powerful CPUs&lt;br /&gt;
* a significant amount of RAM&lt;br /&gt;
&lt;br /&gt;
In fact, only processes run via SLURM have access to all the resources of Mufasa. Processes run outside SLURM are executed by the [[System#Login server|login server]] virtual machine, which has minimal resources and no GPUs. Using SLURM is therefore the only way to execute resource-heavy jobs on Mufasa (this is a key difference between Mufasa 1.0 and Mufasa 2.0).&lt;br /&gt;
&lt;br /&gt;
= SLURM in a nutshell =&lt;br /&gt;
&lt;br /&gt;
Computation jobs on Mufasa needs to be launched via [[System#The SLURM job scheduling system|SLURM]]. SLURM provides jobs with access to the [[#System resources subjected to limitations|physical resources]] of Mufasa, such as CPUs, GPUs and RAM. Thanks to SLURM, processing jobs share system resources, optimising their occupation and availability. &lt;br /&gt;
&lt;br /&gt;
When a user runs a job, the job does not get executed immediately and is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039;. SLURM executes jobs according to their order in the queue: the top job in the queue gets executed as soon as the necessary resources are available, while jobs lower in the queue wait longer. The position of a job in the queue is due to the &amp;#039;&amp;#039;&amp;#039;[[#Job priority|priority]]&amp;#039;&amp;#039;&amp;#039; assigned to it by SLURM, with higher-priority jobs closer to the top. As a general rule:&lt;br /&gt;
&lt;br /&gt;
;: &amp;#039;&amp;#039;&amp;#039;the greater the fraction of Mufasa&amp;#039;s overall resources that a job asks for, the lower the job&amp;#039;s priority will be&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
The priority mechanism is used to encourage users to use Mufasa&amp;#039;s resources (i.e.: GPUs, CPUs, RAM, execution time) in an effective and equitable manner. This page includes a [[#How_to_maximise_the_priority_of_your_jobs|chart explaining how to maximise the priority of your jobs]].&lt;br /&gt;
&lt;br /&gt;
The &amp;#039;&amp;#039;&amp;#039;time&amp;#039;&amp;#039;&amp;#039; available to a job for its execution is controlled by SLURM. When a user requests execution of a job, they must specify the duration of the time slot that the job needs. The job must complete its execution before the end of the requested time slot, because at the end of such slot it gets killed by SLURM.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0 access to system resources is managed via SLURM&amp;#039;s [[#SLURM Quality of Service (QOS)|Quality of Service (QOS)]] mechanism (Mufasa 1.0 used [[#SLURM_partitions|partitions]] instead). While [[User Jobs#Running jobs with SLURM|launching a processing job via SLURM]], the user must always specify the chosen QOS. QOSes differ in the set of resources that they provide access to because each of them is designed to fit a given type of job.&lt;br /&gt;
&lt;br /&gt;
Mufasa sets limits to the number of jobs by the same user. This page includes a [[#Limits on jobs by the same user|table summarising such limits]].&lt;br /&gt;
&lt;br /&gt;
= SLURM Quality of Service (QOS) =&lt;br /&gt;
&lt;br /&gt;
Through &amp;#039;&amp;#039;&amp;#039;Quality of Services&amp;#039;&amp;#039;&amp;#039; (&amp;#039;&amp;#039;&amp;#039;QOSes&amp;#039;&amp;#039;&amp;#039;), SLURM lets system configurators assign a name to a set of related constraints.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, QOSes are used to define different levels of access to the server&amp;#039;s resources. When [[User Jobs|executing a job with SLURM]], a user must always &amp;#039;&amp;#039;&amp;#039;specify the QOS&amp;#039;&amp;#039;&amp;#039; that their job will use: this choice, in turn, determines what resources the job is able to access and influences the [[#Job priority|priority]] of the job.&lt;br /&gt;
&lt;br /&gt;
Mufasa&amp;#039;s QOSes and their features can be inspected with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list qos format=name%-11,priority,MaxSubmit,maxwall,maxtres%-80,mintres%-18&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following (see below for the same data in table form):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
Name          Priority MaxSubmit     MaxWall MaxTRES                                                                          MinTRES            &lt;br /&gt;
----------- ---------- --------- ----------- -------------------------------------------------------------------------------- ------------------ &lt;br /&gt;
normal               0                                                                                                                           &lt;br /&gt;
nogpu                4         1  3-00:00:00 cpu=16,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=128G                    &lt;br /&gt;
gpuheavy-20          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=2,mem=128G             gres/gpu:4g.20gb=1 &lt;br /&gt;
gpuheavy-40          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=1,gres/gpu:4g.20gb=0,mem=128G             gres/gpu:40gb=1    &lt;br /&gt;
gpulight             8         1    12:00:00 cpu=2,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpu                  2         1  1-00:00:00 cpu=8,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpuwide              2         2  1-00:00:00 cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=1,mem=64G              gres/gpu:4g.20gb=1 &lt;br /&gt;
build               32         1    02:00:00 cpu=2,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=16G&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The columns of this output are the following:&lt;br /&gt;
&lt;br /&gt;
:; Name&lt;br /&gt;
:: name of the QOS&lt;br /&gt;
&lt;br /&gt;
:; Priority&lt;br /&gt;
:: priority tier associated to the QOS (higher value = higher priority): see [[#Job priority|Job priority]] for details&lt;br /&gt;
&lt;br /&gt;
:; MaxSubmit&lt;br /&gt;
:: maximum number of jobs from a single user that can be submitted to SLURM with this QOS; submitted jobs include both running and queued jobs&lt;br /&gt;
:: See [[#Limits on jobs by the same user|Limits on jobs by the same user]] for an overview of the limits on jobs set by Mufasa.&lt;br /&gt;
&lt;br /&gt;
:; MaxWall&lt;br /&gt;
:: maximum wall clock duration of the jobs using the QOS (after which they are killed by SLURM), in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
:: For some QOSes these are not set: it means that they are determined by the [[#SLURM partitions|partition]]. Partitions also define the [[#Default values|default duration]] of jobs.&lt;br /&gt;
&lt;br /&gt;
:; MaxTRES&lt;br /&gt;
:: amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) available to a job using the QOS, where&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;cpu=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of CPUs (i.e., processor cores) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of CPUs specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;gres/&amp;#039;&amp;#039;gpu:Type&amp;#039;&amp;#039;=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of GPUs of class &amp;lt;code&amp;gt;&amp;#039;&amp;#039;Type&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; (see [[User Jobs#gres syntax|&amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax]]) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; (for QOSes that allow access to GPUs) if not specified, the job cannot be launched&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;mem=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;G&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum amount of system RAM is &amp;#039;&amp;#039;K&amp;#039;&amp;#039; GBytes&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of RAM specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
&lt;br /&gt;
:; MinTRES&lt;br /&gt;
:: minimum amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) that a job using the QOS must request in order to actually get executed by SLURM.&lt;br /&gt;
:: (If your job does not actually need these resources, you&amp;#039;ve chosen the wrong QOS: you can use one with a higher priority.)&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; QOS is the default one, and exists only to ensure that users always specify a QOS when running a job. Since &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; has zero priority and no resources, a job run using this QOS would never be run.&lt;br /&gt;
&lt;br /&gt;
The information provided by the &amp;lt;code&amp;gt;sacctmgr list qos&amp;lt;/code&amp;gt; command above is summarised by the following table:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! QOS&lt;br /&gt;
! Priority&amp;lt;/br&amp;gt;tier&lt;br /&gt;
! Max&amp;lt;/br&amp;gt;Submit&lt;br /&gt;
! MaxWall&amp;lt;/br&amp;gt;[h]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;CPUs&lt;br /&gt;
! max RAM&amp;lt;/br&amp;gt;[GB]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;3g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;4g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;40GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | build&lt;br /&gt;
| 32&lt;br /&gt;
| 1&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 16&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpulight&lt;br /&gt;
| 8&lt;br /&gt;
| 1&lt;br /&gt;
| 12&lt;br /&gt;
| 2&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | nogpu&lt;br /&gt;
| 4&lt;br /&gt;
| 1&lt;br /&gt;
| 72&lt;br /&gt;
| 16&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpu&lt;br /&gt;
| 2&lt;br /&gt;
| 1&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuwide&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-20&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| 2 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-40&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
A key piece of information in the table above is the &amp;#039;&amp;#039;&amp;#039;priority tier&amp;#039;&amp;#039;&amp;#039; associated to each QOS. In Mufasa 2.0, priority tiers are used to encourage users to use the &amp;#039;&amp;#039;&amp;#039;least powerful QOS that is compatible with their needs&amp;#039;&amp;#039;&amp;#039;, where &amp;quot;powerful&amp;quot; means &amp;quot;rich with resources&amp;quot;. Less powerful QOSes increase the priority of the jobs that use them, so these jobs tend to be executed sooner. See [[#Job priority|Job priority]] for details about how priority affects the execution order of jobs in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
== The &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS ==&lt;br /&gt;
&lt;br /&gt;
This QOS is specifically designed to be used by Mufasa users to &amp;#039;&amp;#039;&amp;#039;quickly build [[System#Containers|container images]]&amp;#039;&amp;#039;&amp;#039;. Its associated priority tier is very high: this, combined with the fact that jobs using this QOS require few resources, means that such jobs usually get executed very soon.&lt;br /&gt;
&lt;br /&gt;
On the other hand, the limited resources and the lack of access to the GPUs make the &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS unsuitable for other tasks.&lt;br /&gt;
&lt;br /&gt;
See [[Singularity#Building Singularity images|Building Singularity images]] for directions about building Singularity container images.&lt;br /&gt;
&lt;br /&gt;
== Restricted QOSes ==&lt;br /&gt;
&lt;br /&gt;
In Mufasa, the most powerful QOSes are reserved to researchers (called &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users: these include academic personnel and Ph.D. students). M.Sc. students, called &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users, cannot use them while running jobs.&lt;br /&gt;
&lt;br /&gt;
See [[#research users and students users|below]] to understand how &amp;#039;&amp;#039;user categories&amp;#039;&amp;#039; work in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
= &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users =&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa belong to two &amp;#039;&amp;#039;&amp;#039;user categories&amp;#039;&amp;#039;&amp;#039;. Categories provide users with different access to  to Mufasa&amp;#039;s resources: the idea being to provide researchers with more access while still letting students benefit from the server&amp;#039;s capabilities.&lt;br /&gt;
&lt;br /&gt;
User categories in Mufasa are the following:&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. academic personnel and Ph.D. students&lt;br /&gt;
::: * can use all QOSes, including the the [[#Restricted QOSes|restricted ones]]&lt;br /&gt;
::: * their jobs have a higher &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a higher number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. M.Sc. students&lt;br /&gt;
::: * cannot use the [[#Restricted QOSes|restricted QOSes]]&lt;br /&gt;
::: * their jobs have a lower &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a lower number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
You can inspect the differences in priority and running jobs between &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association format=&amp;quot;account,priority,maxjobs&amp;quot; | grep -E &amp;#039;Account|research|students&amp;#039;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
   Account   Priority MaxJobs &lt;br /&gt;
  research          4       2 &lt;br /&gt;
  students          1       1&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
To know what limits apply to your own user, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association where user=$USER format=&amp;quot;user,priority,maxjobs,qos%-60&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
      User   Priority MaxJobs QOS                                                          &lt;br /&gt;
---------- ---------- ------- ------------------------------------------------------------ &lt;br /&gt;
    preali          4       2 build,gpu,gpuheavy-20,gpuheavy-40,gpulight,gpuwide,nogpu&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The list under &amp;quot;QOS&amp;quot; shows what QOSes your user is allowed to use when [[User Jobs|running jobs]]. &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users can use all of them, while &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users can only access a subset of them.&lt;br /&gt;
&lt;br /&gt;
= Limits on jobs by the same user =&lt;br /&gt;
&lt;br /&gt;
Mufasa enforces limits on the number of jobs from a single user. Such limits aim at preventing users from &amp;quot;hogging&amp;quot; system resources, and apply to:&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;submitted jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that the user asked SLURM to execute, each of which may currently be either running or queued&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;running jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that are currently in execution&lt;br /&gt;
&lt;br /&gt;
Limits on submitted jobs are set via [[#SLURM Quality of Service (QOS)|QOSes]], while limits on running jobs are set via [[#research users and students users|user category]].&lt;br /&gt;
&lt;br /&gt;
The following table summarises what limits exist:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! &lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;running&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;submitted&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | global limits&amp;lt;br/&amp;gt;(system-wide)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to limits set by individual QOSes (below)&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | limits for&amp;lt;br/&amp;gt;individual&amp;lt;/br&amp;gt;QOSes&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to global limits (above)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for the&amp;#039;&amp;#039; &amp;lt;code&amp;gt;gpuwide&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;QOS&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for each of the other QOSes&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
= Job priority =&lt;br /&gt;
&lt;br /&gt;
Once the execution of a job has been requested, the job is not run immediately: it is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039; by SLURM, together with all the other jobs awaiting execution. The job on top of the queue at any time is the first to be put into execution as soon as the resources it requires are available. &lt;br /&gt;
&lt;br /&gt;
The order of the items in the job queue depends on their &amp;#039;&amp;#039;&amp;#039;priority&amp;#039;&amp;#039;&amp;#039;. The job with the highest priority is put by SLURM on top of the queue, while all other jobs are put in the queue in order of descending priority.&lt;br /&gt;
&lt;br /&gt;
The goal of SLURM is to maximise resource availability: i.e., to ensure the shorter possible wait time before job execution. To achieve this goal, in Mufasa SLURM is configured to &amp;#039;&amp;#039;&amp;#039;encourage users not to ask for resources or execution time that their job doesn&amp;#039;t need&amp;#039;&amp;#039;&amp;#039;. This is done via the priority mechanism: the more resources and/or time a job requests, the lower its priority will be; and the later it will be executed.&lt;br /&gt;
&lt;br /&gt;
Priority management in Mufasa is designed to set up a &amp;#039;&amp;#039;&amp;#039;virtuous cycle&amp;#039;&amp;#039;&amp;#039; where users, by carefully choosing what to ask for, obtain two results:&lt;br /&gt;
* they ensure that their job is executed as soon as possible;&lt;br /&gt;
* they leave as much as possible of Mufasa&amp;#039;s resources free for other users&amp;#039;s jobs.&lt;br /&gt;
&lt;br /&gt;
== Elements determining job priority ==&lt;br /&gt;
In Mufasa, the priority of a job is computed by SLURM according to the following elements:&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#research users and students users|User category]]&amp;#039;&amp;#039;&amp;#039; (i.e., &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;)&lt;br /&gt;
::: Used to provide higher priority to jobs run by &amp;#039;&amp;#039;&amp;#039;research personnel&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#SLURM Quality of Service (QOS)|QOS]]&amp;#039;&amp;#039;&amp;#039; used by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;less resources&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Number of CPUs&amp;#039;&amp;#039;&amp;#039; requested by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;a lower number of CPUs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job Age&amp;#039;&amp;#039;&amp;#039;, i.e. the time that the job has been waiting in the queue&lt;br /&gt;
::: Used to provide higher priority to jobs which have been &amp;#039;&amp;#039;&amp;#039;queued for a longer time&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
::: Note: periods when a job is prevented from running by the [[#Limits on jobs by the same user|limits on running jobs by the same user]] do not increase job age.&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;FairShare&amp;#039;&amp;#039;&amp;#039;, i.e. a factor computed by SLURM to balance use of the system by different users&lt;br /&gt;
::: Used to provide higher priority to jobs by users who &amp;#039;&amp;#039;&amp;#039;used Mufasa&amp;#039;s resources less than others&amp;#039;&amp;#039;&amp;#039; (see [[#How FairShare works|below]])&lt;br /&gt;
&lt;br /&gt;
You can understand how each of these elements contributes to the priority of the jobs waiting for execution (both yours and of other users) with command &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;. A tailored version of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; that only displays the parameters listed above is the following:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sprio -o &amp;quot;%6i %9u %.8Y %.10B %.5Q %.9T %.5A %.5F&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Here is an example of the output of the command above:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
JOBID  USER      PRIORITY      ASSOC   QOS      TRES   AGE FAIRS&lt;br /&gt;
5790   amoglia       3495       2500   375  cpu=-800   933   487&lt;br /&gt;
5791   qliu          3010       2500   375  cpu=-800   916    19&lt;br /&gt;
5817   mzoccola      2336        625   750  cpu=-800   495  1266&lt;br /&gt;
5822   fbottazzi     2235        625   750  cpu=-600   467   994&lt;br /&gt;
5837   lpassoni      2508       2500   750  cpu=-800     0    58&lt;br /&gt;
5838   bagostini     4437        625  3000  cpu=-200     0  1013&lt;br /&gt;
5839   flazzaro      1549        625   750  cpu=-800     0   974&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
In it, column PRIORITY provides the overall priority of each job. Each priority value is the sum of the five values to its right; in particular, column ASSOC corresponds to user category, while column TRES corresponds to the number of CPUs. Note that the contribution of TRES to priority is &amp;#039;&amp;#039;negative&amp;#039;&amp;#039;, i.e. it is a penalty; its value is proportional to the number of CPUs requested by the job.&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;Things to consider when using &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
* Its output is only relevant for jobs &amp;#039;&amp;#039;not yet in execution&amp;#039;&amp;#039;. For an already running job, the values shown by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; correspond to the priority that the job would get if evaluated now, which generally differs from its actual priority.&lt;br /&gt;
* Jobs listed by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; may temporarily differ from jobs listed by &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;: this is due to the fact that SLURM updates its internal data with a periodicity of a few minutes.&lt;br /&gt;
&lt;br /&gt;
=== How FairShare works ===&lt;br /&gt;
In Mufasa 2.0, the FairShare of each user is computed according to these rules:&lt;br /&gt;
# It considers the quantity used by (current and past) user jobs of:&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;CPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of CPUs used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;RAM&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the amount of RAM used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;GPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of GPUs used&lt;br /&gt;
#** note: 40 GB GPUs have double the impact on FairShare than 20 GB GPUs&lt;br /&gt;
# The impact on FairShare of any use of resources is &amp;#039;&amp;#039;&amp;#039;proportional to its duration&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
#* example: using 32 GB of RAM for 48h has the same impact of using 64 GB of RAM for 24h&lt;br /&gt;
# FairShare has a &amp;quot;fading memory&amp;quot;: i.e., resource use has less and less impact on FairShare the farther in the past it is&lt;br /&gt;
#* consequence: over time, the &amp;quot;history&amp;quot; of a user gets forgotten by FairShare&lt;br /&gt;
&lt;br /&gt;
== How to maximise the priority of your jobs ==&lt;br /&gt;
&lt;br /&gt;
Every time you run a SLURM job, follow these guidelines:&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Choose the less powerful QOS compatible with the needs of your job&lt;br /&gt;
:: QOSes with access to less resources have higher base priority&lt;br /&gt;
&lt;br /&gt;
; Only request CPUs and RAM that your job really needs&lt;br /&gt;
:: Usually, code exploits multiple CPUs only if specifically designed to do so: if unsure, only ask for 1 CPU&lt;br /&gt;
:: The fewer resources your job asks for, the less it will wait before they become available&lt;br /&gt;
:: The more CPUs your job asks for, the stronger the penalty to its priority&lt;br /&gt;
:: Asking for fewer CPUs and/or less RAM improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Do not request more time than your jobs needs to complete&lt;br /&gt;
:: Make a worst-case estimate and only ask for that duration&lt;br /&gt;
:: Asking for less of Mufasa&amp;#039;s time improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Debug and test code using less powerful QOSes before running it with more powerful QOSes&lt;br /&gt;
:: Your test jobs will get a higher priority and your FairShare will improve&lt;br /&gt;
&lt;br /&gt;
; Cancel jobs that you don&amp;#039;t need them anymore&lt;br /&gt;
:: Always use [[User_Jobs#Cancelling_a_job_with_scancel|scancel]] to delete completed (or crashed) jobs: your Fairshare will improve&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Protip: it&amp;#039;s a good idea to [[User Jobs#Looking for unused GPUs|check for unused GPUs]] before choosing what to request. Requesting a GPU that is currently idle will help your job get executed sooner.&lt;br /&gt;
&lt;br /&gt;
= SLURM partitions =&lt;br /&gt;
&lt;br /&gt;
Partitions are another mechanism provided by SLURM to create different levels of access to system resources. Since in Mufasa 2.0 access to resources is controlled via [[#SLURM Quality of Service (QOS)|QOSes]], partitions are not very relevant. (This is one of the main differences between Mufasa 1.0 and Mufasa 2.0.)&lt;br /&gt;
&lt;br /&gt;
Note, however, that the default values for some features of SLURM jobs (e.g., duration) are [[#Default values|set by the partition]].&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, there is a single SLURM partition, called &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt;, and all jobs run on it. The state of &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt; can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sinfo -o &amp;quot;%10P %5a %9T %11L %10l&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PARTITION  AVAIL STATE     DEFAULTTIME TIMELIMIT &lt;br /&gt;
jobs*      up    idle      1:00:00     3-00:00:00&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where columns correspond to the following information:&lt;br /&gt;
&lt;br /&gt;
:; PARTITION&lt;br /&gt;
:: name of the partition; the asterisks indicates that it&amp;#039;s the default one&lt;br /&gt;
&lt;br /&gt;
:; AVAIL&lt;br /&gt;
:: state/availability of the partition: see [[#Partition availability|below]]&lt;br /&gt;
&lt;br /&gt;
:; STATE&lt;br /&gt;
:: state (using [https://slurm.schedmd.com/sinfo.html#SECTION_NODE-STATE-CODES these codes])&lt;br /&gt;
:: typical values are &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;mixed&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that some of the resources are busy executing jobs while other are idle, and &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;allocated&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that all of the resources are in use&lt;br /&gt;
&lt;br /&gt;
:; DEFAULTTIME&lt;br /&gt;
:: default runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:; TIMELIMIT&lt;br /&gt;
:: maximum runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
The asterisk at the end of the partition name indicates the default partition, i.e. the one on which jobs which do not ask for a specific partition are run.&lt;br /&gt;
&lt;br /&gt;
Command &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; does not tell you about the &amp;#039;&amp;#039;jobs&amp;#039;&amp;#039; submitted to a partition. This information is obtained, instead, with [[User Jobs#Inspecting jobs with squeue|command &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;]].&lt;br /&gt;
&lt;br /&gt;
== Partition availability ==&lt;br /&gt;
&lt;br /&gt;
The most important information that &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; provides is the &amp;#039;&amp;#039;&amp;#039;availability&amp;#039;&amp;#039;&amp;#039; (also called &amp;#039;&amp;#039;&amp;#039;state&amp;#039;&amp;#039;&amp;#039;) of partitions. This is shown in column &amp;quot;AVAIL&amp;quot;. Possible partition states are:&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is available&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed as soon as resources allow&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is in the process of becoming unavailable (i.e., of entering the &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; state: see below)&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is unavailable&lt;br /&gt;
:: There are no running jobs&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
When a partition goes from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; no harm is done to running jobs. In a normally functioning SLURM system, the passage from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; happens only when no jobs are running on the partition. If (e.g., due to a malfunction) the passage happens with jobs still running, they get killed.&lt;br /&gt;
&lt;br /&gt;
A partition in state &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; requires intervention by a [[Roles|Job Administrator]] to be restored to &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
== Default values ==&lt;br /&gt;
&lt;br /&gt;
The features of SLURM partitions, including the &amp;#039;&amp;#039;&amp;#039;default values&amp;#039;&amp;#039;&amp;#039; which are applied to jobs that do not make explicit requests, can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scontrol show partition&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PartitionName=jobs&lt;br /&gt;
   AllowGroups=ALL AllowAccounts=ALL AllowQos=nogpu,gpulight,gpu,gpuwide,gpuheavy-20,gpuheavy-40&lt;br /&gt;
   AllocNodes=ALL Default=YES QoS=N/A&lt;br /&gt;
   DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO&lt;br /&gt;
   MaxNodes=UNLIMITED MaxTime=3-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED&lt;br /&gt;
   Nodes=gn01&lt;br /&gt;
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO&lt;br /&gt;
   OverTimeLimit=NONE PreemptMode=OFF&lt;br /&gt;
   State=UP TotalCPUs=48 TotalNodes=1 SelectTypeParameters=NONE&lt;br /&gt;
   JobDefaults=(null)&lt;br /&gt;
   DefMemPerNode=4096 MaxMemPerNode=UNLIMITED&lt;br /&gt;
   TRES=cpu=48,mem=1011435M,node=1,billing=49,gres/gpu=13,gres/gpu:3g.20gb=5,gres/gpu:40gb=3,gres/gpu:4g.20gb=5&lt;br /&gt;
   TRESBillingWeights=cpu=1.0,gres/gpu:3g.20gb=6.0,gres/gpu:4g.20gb=6.0,gres/gpu:40gb=6.0,mem=0.05g&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In the output of &amp;lt;code&amp;gt;scontrol show partition&amp;lt;/code&amp;gt; the most relevant default values for Mufasa users are:&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefaultTime&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default execution time assigned to a job run on the partition (e.g., 1 hour)&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefMemPerNode&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default amount of RAM assigned to a job run on the partition (e.g., 4GB)&lt;br /&gt;
&lt;br /&gt;
= System resources subjected to limitations =&lt;br /&gt;
&lt;br /&gt;
In systems based on SLURM like Mufasa, &amp;#039;&amp;#039;&amp;#039;TRES (Trackable RESources)&amp;#039;&amp;#039;&amp;#039; are (from [https://slurm.schedmd.com/tres.html SLURM&amp;#039;s documentation] &amp;quot;&amp;#039;&amp;#039;resources that can be tracked for usage or used to enforce limits against.&amp;#039;&amp;#039;&amp;quot;&lt;br /&gt;
&lt;br /&gt;
TRES include CPUs, RAM and &amp;#039;&amp;#039;&amp;#039;GRES&amp;#039;&amp;#039;&amp;#039;. The last term stands for &amp;#039;&amp;#039;Generic RESources&amp;#039;&amp;#039; that a job may need for its execution. In Mufasa, the only &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; resources are the GPUs.&lt;br /&gt;
&lt;br /&gt;
== &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax ==&lt;br /&gt;
&lt;br /&gt;
To ask SLURM to assign GRES resources (i.e., GPUs) to a job, a special syntax must be used. Precisely, the name of each GPU resource takes the form&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:Name:Type&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
Considering the [[System#CPUs and GPUs|GPU complement of Mufasa]], &amp;lt;code&amp;gt;Type&amp;lt;/code&amp;gt; takes the following values:&lt;br /&gt;
&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:40gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 40 Gbytes of RAM&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:4g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 4 compute units&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 3 compute units&lt;br /&gt;
&lt;br /&gt;
So, for instance,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
identifies a resource corresponding to a GPU with 20 GB of RAM and 3 compute units.&lt;br /&gt;
&lt;br /&gt;
When asking for a GRES resource (e.g., in an &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command or an &amp;lt;code&amp;gt;SBATCH&amp;lt;/code&amp;gt; directive of an [[User Jobs#Using execution scripts to run jobs|execution script]]), the syntax required by SLURM is&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:&amp;lt;Type&amp;gt;:&amp;lt;Quantity&amp;gt;&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
where &amp;lt;code&amp;gt;Quantity&amp;lt;/code&amp;gt; is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of type &amp;lt;code&amp;gt;4g.20gb&amp;lt;/code&amp;gt; the syntax is&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:4g.20gb:2&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SLURM&amp;#039;s &amp;#039;&amp;#039;generic resources&amp;#039;&amp;#039; are defined in &amp;lt;code&amp;gt;/etc/slurm/gres.conf&amp;lt;/code&amp;gt;. In order to make GPUs available to SLURM&amp;#039;s &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; management, Mufasa makes use of Nvidia&amp;#039;s [https://developer.nvidia.com/nvidia-management-library-nvml NVML library]. For additional information see [https://slurm.schedmd.com/gres.html SLURM&amp;#039;s documentation].&lt;/div&gt;</summary>
		<author><name>GiulioFontana</name></author>
	</entry>
	<entry>
		<id>https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2595</id>
		<title>SLURM</title>
		<link rel="alternate" type="text/html" href="https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2595"/>
		<updated>2026-11-18T09:02:29Z</updated>

		<summary type="html">&lt;p&gt;GiulioFontana: /* Elements determining job priority */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page presents the features of SLURM that are most relevant to Mufasa&amp;#039;s [[Roles|Job Users]]. Job Users can submit jobs for execution, cancel their own jobs, and see other users&amp;#039; jobs (but not intervene on them).&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa &amp;#039;&amp;#039;&amp;#039;must use SLURM&amp;#039;&amp;#039;&amp;#039; to run resource-heavy processes, i.e. computing jobs that require one or more of the following:&lt;br /&gt;
* GPUs&lt;br /&gt;
* multiple CPUs&lt;br /&gt;
* powerful CPUs&lt;br /&gt;
* a significant amount of RAM&lt;br /&gt;
&lt;br /&gt;
In fact, only processes run via SLURM have access to all the resources of Mufasa. Processes run outside SLURM are executed by the [[System#Login server|login server]] virtual machine, which has minimal resources and no GPUs. Using SLURM is therefore the only way to execute resource-heavy jobs on Mufasa (this is a key difference between Mufasa 1.0 and Mufasa 2.0).&lt;br /&gt;
&lt;br /&gt;
= SLURM in a nutshell =&lt;br /&gt;
&lt;br /&gt;
Computation jobs on Mufasa needs to be launched via [[System#The SLURM job scheduling system|SLURM]]. SLURM provides jobs with access to the [[#System resources subjected to limitations|physical resources]] of Mufasa, such as CPUs, GPUs and RAM. Thanks to SLURM, processing jobs share system resources, optimising their occupation and availability. &lt;br /&gt;
&lt;br /&gt;
When a user runs a job, the job does not get executed immediately and is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039;. SLURM executes jobs according to their order in the queue: the top job in the queue gets executed as soon as the necessary resources are available, while jobs lower in the queue wait longer. The position of a job in the queue is due to the &amp;#039;&amp;#039;&amp;#039;[[#Job priority|priority]]&amp;#039;&amp;#039;&amp;#039; assigned to it by SLURM, with higher-priority jobs closer to the top. As a general rule:&lt;br /&gt;
&lt;br /&gt;
;: &amp;#039;&amp;#039;&amp;#039;the greater the fraction of Mufasa&amp;#039;s overall resources that a job asks for, the lower the job&amp;#039;s priority will be&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
The priority mechanism is used to encourage users to use Mufasa&amp;#039;s resources (i.e.: GPUs, CPUs, RAM, execution time) in an effective and equitable manner. This page includes a [[#How_to_maximise_the_priority_of_your_jobs|chart explaining how to maximise the priority of your jobs]].&lt;br /&gt;
&lt;br /&gt;
The &amp;#039;&amp;#039;&amp;#039;time&amp;#039;&amp;#039;&amp;#039; available to a job for its execution is controlled by SLURM. When a user requests execution of a job, they must specify the duration of the time slot that the job needs. The job must complete its execution before the end of the requested time slot, because at the end of such slot it gets killed by SLURM.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0 access to system resources is managed via SLURM&amp;#039;s [[#SLURM Quality of Service (QOS)|Quality of Service (QOS)]] mechanism (Mufasa 1.0 used [[#SLURM_partitions|partitions]] instead). While [[User Jobs#Running jobs with SLURM|launching a processing job via SLURM]], the user must always specify the chosen QOS. QOSes differ in the set of resources that they provide access to because each of them is designed to fit a given type of job.&lt;br /&gt;
&lt;br /&gt;
Mufasa sets limits to the number of jobs by the same user. This page includes a [[#Limits on jobs by the same user|table summarising such limits]].&lt;br /&gt;
&lt;br /&gt;
= SLURM Quality of Service (QOS) =&lt;br /&gt;
&lt;br /&gt;
Through &amp;#039;&amp;#039;&amp;#039;Quality of Services&amp;#039;&amp;#039;&amp;#039; (&amp;#039;&amp;#039;&amp;#039;QOSes&amp;#039;&amp;#039;&amp;#039;), SLURM lets system configurators assign a name to a set of related constraints.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, QOSes are used to define different levels of access to the server&amp;#039;s resources. When [[User Jobs|executing a job with SLURM]], a user must always &amp;#039;&amp;#039;&amp;#039;specify the QOS&amp;#039;&amp;#039;&amp;#039; that their job will use: this choice, in turn, determines what resources the job is able to access and influences the [[#Job priority|priority]] of the job.&lt;br /&gt;
&lt;br /&gt;
Mufasa&amp;#039;s QOSes and their features can be inspected with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list qos format=name%-11,priority,MaxSubmit,maxwall,maxtres%-80,mintres%-18&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following (see below for the same data in table form):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
Name          Priority MaxSubmit     MaxWall MaxTRES                                                                          MinTRES            &lt;br /&gt;
----------- ---------- --------- ----------- -------------------------------------------------------------------------------- ------------------ &lt;br /&gt;
normal               0                                                                                                                           &lt;br /&gt;
nogpu                4         1  3-00:00:00 cpu=16,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=128G                    &lt;br /&gt;
gpuheavy-20          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=2,mem=128G             gres/gpu:4g.20gb=1 &lt;br /&gt;
gpuheavy-40          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=1,gres/gpu:4g.20gb=0,mem=128G             gres/gpu:40gb=1    &lt;br /&gt;
gpulight             8         1    12:00:00 cpu=2,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpu                  2         1  1-00:00:00 cpu=8,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpuwide              2         2  1-00:00:00 cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=1,mem=64G              gres/gpu:4g.20gb=1 &lt;br /&gt;
build               32         1    02:00:00 cpu=2,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=16G&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The columns of this output are the following:&lt;br /&gt;
&lt;br /&gt;
:; Name&lt;br /&gt;
:: name of the QOS&lt;br /&gt;
&lt;br /&gt;
:; Priority&lt;br /&gt;
:: priority tier associated to the QOS (higher value = higher priority): see [[#Job priority|Job priority]] for details&lt;br /&gt;
&lt;br /&gt;
:; MaxSubmit&lt;br /&gt;
:: maximum number of jobs from a single user that can be submitted to SLURM with this QOS; submitted jobs include both running and queued jobs&lt;br /&gt;
:: See [[#Limits on jobs by the same user|Limits on jobs by the same user]] for an overview of the limits on jobs set by Mufasa.&lt;br /&gt;
&lt;br /&gt;
:; MaxWall&lt;br /&gt;
:: maximum wall clock duration of the jobs using the QOS (after which they are killed by SLURM), in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
:: For some QOSes these are not set: it means that they are determined by the [[#SLURM partitions|partition]]. Partitions also define the [[#Default values|default duration]] of jobs.&lt;br /&gt;
&lt;br /&gt;
:; MaxTRES&lt;br /&gt;
:: amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) available to a job using the QOS, where&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;cpu=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of CPUs (i.e., processor cores) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of CPUs specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;gres/&amp;#039;&amp;#039;gpu:Type&amp;#039;&amp;#039;=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of GPUs of class &amp;lt;code&amp;gt;&amp;#039;&amp;#039;Type&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; (see [[User Jobs#gres syntax|&amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax]]) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; (for QOSes that allow access to GPUs) if not specified, the job cannot be launched&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;mem=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;G&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum amount of system RAM is &amp;#039;&amp;#039;K&amp;#039;&amp;#039; GBytes&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of RAM specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
&lt;br /&gt;
:; MinTRES&lt;br /&gt;
:: minimum amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) that a job using the QOS must request in order to actually get executed by SLURM.&lt;br /&gt;
:: (If your job does not actually need these resources, you&amp;#039;ve chosen the wrong QOS: you can use one with a higher priority.)&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; QOS is the default one, and exists only to ensure that users always specify a QOS when running a job. Since &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; has zero priority and no resources, a job run using this QOS would never be run.&lt;br /&gt;
&lt;br /&gt;
The information provided by the &amp;lt;code&amp;gt;sacctmgr list qos&amp;lt;/code&amp;gt; command above is summarised by the following table:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! QOS&lt;br /&gt;
! Priority&amp;lt;/br&amp;gt;tier&lt;br /&gt;
! Max&amp;lt;/br&amp;gt;Submit&lt;br /&gt;
! MaxWall&amp;lt;/br&amp;gt;[h]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;CPUs&lt;br /&gt;
! max RAM&amp;lt;/br&amp;gt;[GB]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;3g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;4g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;40GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | build&lt;br /&gt;
| 32&lt;br /&gt;
| 1&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 16&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpulight&lt;br /&gt;
| 8&lt;br /&gt;
| 1&lt;br /&gt;
| 12&lt;br /&gt;
| 2&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | nogpu&lt;br /&gt;
| 4&lt;br /&gt;
| 1&lt;br /&gt;
| 72&lt;br /&gt;
| 16&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpu&lt;br /&gt;
| 2&lt;br /&gt;
| 1&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuwide&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-20&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| 2 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-40&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
A key piece of information in the table above is the &amp;#039;&amp;#039;&amp;#039;priority tier&amp;#039;&amp;#039;&amp;#039; associated to each QOS. In Mufasa 2.0, priority tiers are used to encourage users to use the &amp;#039;&amp;#039;&amp;#039;least powerful QOS that is compatible with their needs&amp;#039;&amp;#039;&amp;#039;, where &amp;quot;powerful&amp;quot; means &amp;quot;rich with resources&amp;quot;. Less powerful QOSes increase the priority of the jobs that use them, so these jobs tend to be executed sooner. See [[#Job priority|Job priority]] for details about how priority affects the execution order of jobs in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
== The &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS ==&lt;br /&gt;
&lt;br /&gt;
This QOS is specifically designed to be used by Mufasa users to &amp;#039;&amp;#039;&amp;#039;quickly build [[System#Containers|container images]]&amp;#039;&amp;#039;&amp;#039;. Its associated priority tier is very high: this, combined with the fact that jobs using this QOS require few resources, means that such jobs usually get executed very soon.&lt;br /&gt;
&lt;br /&gt;
On the other hand, the limited resources and the lack of access to the GPUs make the &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS unsuitable for other tasks.&lt;br /&gt;
&lt;br /&gt;
See [[Singularity#Building Singularity images|Building Singularity images]] for directions about building Singularity container images.&lt;br /&gt;
&lt;br /&gt;
== Restricted QOSes ==&lt;br /&gt;
&lt;br /&gt;
In Mufasa, the most powerful QOSes are reserved to researchers (called &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users: these include academic personnel and Ph.D. students). M.Sc. students, called &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users, cannot use them while running jobs.&lt;br /&gt;
&lt;br /&gt;
See [[#research users and students users|below]] to understand how &amp;#039;&amp;#039;user categories&amp;#039;&amp;#039; work in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
= &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users =&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa belong to two &amp;#039;&amp;#039;&amp;#039;user categories&amp;#039;&amp;#039;&amp;#039;. Categories provide users with different access to  to Mufasa&amp;#039;s resources: the idea being to provide researchers with more access while still letting students benefit from the server&amp;#039;s capabilities.&lt;br /&gt;
&lt;br /&gt;
User categories in Mufasa are the following:&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. academic personnel and Ph.D. students&lt;br /&gt;
::: * can use all QOSes, including the the [[#Restricted QOSes|restricted ones]]&lt;br /&gt;
::: * their jobs have a higher &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a higher number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. M.Sc. students&lt;br /&gt;
::: * cannot use the [[#Restricted QOSes|restricted QOSes]]&lt;br /&gt;
::: * their jobs have a lower &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a lower number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
You can inspect the differences in priority and running jobs between &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association format=&amp;quot;account,priority,maxjobs&amp;quot; | grep -E &amp;#039;Account|research|students&amp;#039;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
   Account   Priority MaxJobs &lt;br /&gt;
  research          4       2 &lt;br /&gt;
  students          1       1&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
To know what limits apply to your own user, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association where user=$USER format=&amp;quot;user,priority,maxjobs,qos%-60&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
      User   Priority MaxJobs QOS                                                          &lt;br /&gt;
---------- ---------- ------- ------------------------------------------------------------ &lt;br /&gt;
    preali          4       2 build,gpu,gpuheavy-20,gpuheavy-40,gpulight,gpuwide,nogpu&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The list under &amp;quot;QOS&amp;quot; shows what QOSes your user is allowed to use when [[User Jobs|running jobs]]. &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users can use all of them, while &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users can only access a subset of them.&lt;br /&gt;
&lt;br /&gt;
= Limits on jobs by the same user =&lt;br /&gt;
&lt;br /&gt;
Mufasa enforces limits on the number of jobs from a single user. Such limits aim at preventing users from &amp;quot;hogging&amp;quot; system resources, and apply to:&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;submitted jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that the user asked SLURM to execute, each of which may currently be either running or queued&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;running jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that are currently in execution&lt;br /&gt;
&lt;br /&gt;
Limits on submitted jobs are set via [[#SLURM Quality of Service (QOS)|QOSes]], while limits on running jobs are set via [[#research users and students users|user category]].&lt;br /&gt;
&lt;br /&gt;
The following table summarises what limits exist:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! &lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;running&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;submitted&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | global limits&amp;lt;br/&amp;gt;(system-wide)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to limits set by individual QOSes (below)&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | limits for&amp;lt;br/&amp;gt;individual&amp;lt;/br&amp;gt;QOSes&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to global limits (above)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for the&amp;#039;&amp;#039; &amp;lt;code&amp;gt;gpuwide&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;QOS&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for each of the other QOSes&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
= Job priority =&lt;br /&gt;
&lt;br /&gt;
Once the execution of a job has been requested, the job is not run immediately: it is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039; by SLURM, together with all the other jobs awaiting execution. The job on top of the queue at any time is the first to be put into execution as soon as the resources it requires are available. &lt;br /&gt;
&lt;br /&gt;
The order of the items in the job queue depends on their &amp;#039;&amp;#039;&amp;#039;priority&amp;#039;&amp;#039;&amp;#039;. The job with the highest priority is put by SLURM on top of the queue, while all other jobs are put in the queue in order of descending priority.&lt;br /&gt;
&lt;br /&gt;
The goal of SLURM is to maximise resource availability: i.e., to ensure the shorter possible wait time before job execution. To achieve this goal, in Mufasa SLURM is configured to &amp;#039;&amp;#039;&amp;#039;encourage users not to ask for resources or execution time that their job doesn&amp;#039;t need&amp;#039;&amp;#039;&amp;#039;. This is done via the priority mechanism: the more resources and/or time a job requests, the lower its priority will be; and the later it will be executed.&lt;br /&gt;
&lt;br /&gt;
Priority management in Mufasa is designed to set up a &amp;#039;&amp;#039;&amp;#039;virtuous cycle&amp;#039;&amp;#039;&amp;#039; where users, by carefully choosing what to ask for, obtain two results:&lt;br /&gt;
* they ensure that their job is executed as soon as possible;&lt;br /&gt;
* they leave as much as possible of Mufasa&amp;#039;s resources free for other users&amp;#039;s jobs.&lt;br /&gt;
&lt;br /&gt;
== Elements determining job priority ==&lt;br /&gt;
In Mufasa, the priority of a job is computed by SLURM according to the following elements:&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#research users and students users|User category]]&amp;#039;&amp;#039;&amp;#039; (i.e., &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;)&lt;br /&gt;
::: Used to provide higher priority to jobs run by &amp;#039;&amp;#039;&amp;#039;research personnel&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#SLURM Quality of Service (QOS)|QOS]]&amp;#039;&amp;#039;&amp;#039; used by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;less resources&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Number of CPUs&amp;#039;&amp;#039;&amp;#039; requested by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;a lower number of CPUs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job Age&amp;#039;&amp;#039;&amp;#039;, i.e. the time that the job has been waiting in the queue&lt;br /&gt;
::: Used to provide higher priority to jobs which have been &amp;#039;&amp;#039;&amp;#039;queued for a longer time&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
::: Note: age is computed only when a queued job is &amp;#039;&amp;#039;eligible for running&amp;#039;&amp;#039;. Time when a job cannot run because of the [[#Limits on jobs by the same user|limits on running jobs by the same user]] does not count.&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;FairShare&amp;#039;&amp;#039;&amp;#039;, i.e. a factor computed by SLURM to balance use of the system by different users&lt;br /&gt;
::: Used to provide higher priority to jobs by users who &amp;#039;&amp;#039;&amp;#039;used Mufasa&amp;#039;s resources less than others&amp;#039;&amp;#039;&amp;#039; (see [[#How FairShare works|below]])&lt;br /&gt;
&lt;br /&gt;
You can understand how each of these elements contributes to the priority of the jobs waiting for execution (both yours and of other users) with command &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;. A tailored version of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; that only displays the parameters listed above is the following:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sprio -o &amp;quot;%6i %9u %.8Y %.10B %.5Q %.9T %.5A %.5F&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Here is an example of the output of the command above:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
JOBID  USER      PRIORITY      ASSOC   QOS      TRES   AGE FAIRS&lt;br /&gt;
5790   amoglia       3495       2500   375  cpu=-800   933   487&lt;br /&gt;
5791   qliu          3010       2500   375  cpu=-800   916    19&lt;br /&gt;
5817   mzoccola      2336        625   750  cpu=-800   495  1266&lt;br /&gt;
5822   fbottazzi     2235        625   750  cpu=-600   467   994&lt;br /&gt;
5837   lpassoni      2508       2500   750  cpu=-800     0    58&lt;br /&gt;
5838   bagostini     4437        625  3000  cpu=-200     0  1013&lt;br /&gt;
5839   flazzaro      1549        625   750  cpu=-800     0   974&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
In it, column PRIORITY provides the overall priority of each job. Each priority value is the sum of the five values to its right; in particular, column ASSOC corresponds to user category, while column TRES corresponds to the number of CPUs. Note that the contribution of TRES to priority is &amp;#039;&amp;#039;negative&amp;#039;&amp;#039;, i.e. it is a penalty; its value is proportional to the number of CPUs requested by the job.&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;Things to consider when using &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
* Its output is only relevant for jobs &amp;#039;&amp;#039;not yet in execution&amp;#039;&amp;#039;. For an already running job, the values shown by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; correspond to the priority that the job would get if evaluated now, which generally differs from its actual priority.&lt;br /&gt;
* Jobs listed by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; may temporarily differ from jobs listed by &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;: this is due to the fact that SLURM updates its internal data with a periodicity of a few minutes.&lt;br /&gt;
&lt;br /&gt;
=== How FairShare works ===&lt;br /&gt;
In Mufasa 2.0, the FairShare of each user is computed according to these rules:&lt;br /&gt;
# It considers the quantity used by (current and past) user jobs of:&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;CPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of CPUs used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;RAM&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the amount of RAM used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;GPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of GPUs used&lt;br /&gt;
#** note: 40 GB GPUs have double the impact on FairShare than 20 GB GPUs&lt;br /&gt;
# The impact on FairShare of any use of resources is &amp;#039;&amp;#039;&amp;#039;proportional to its duration&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
#* example: using 32 GB of RAM for 48h has the same impact of using 64 GB of RAM for 24h&lt;br /&gt;
# FairShare has a &amp;quot;fading memory&amp;quot;: i.e., resource use has less and less impact on FairShare the farther in the past it is&lt;br /&gt;
#* consequence: over time, the &amp;quot;history&amp;quot; of a user gets forgotten by FairShare&lt;br /&gt;
&lt;br /&gt;
== How to maximise the priority of your jobs ==&lt;br /&gt;
&lt;br /&gt;
Every time you run a SLURM job, follow these guidelines:&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Choose the less powerful QOS compatible with the needs of your job&lt;br /&gt;
:: QOSes with access to less resources have higher base priority&lt;br /&gt;
&lt;br /&gt;
; Only request CPUs and RAM that your job really needs&lt;br /&gt;
:: Usually, code exploits multiple CPUs only if specifically designed to do so: if unsure, only ask for 1 CPU&lt;br /&gt;
:: The fewer resources your job asks for, the less it will wait before they become available&lt;br /&gt;
:: The more CPUs your job asks for, the stronger the penalty to its priority&lt;br /&gt;
:: Asking for fewer CPUs and/or less RAM improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Do not request more time than your jobs needs to complete&lt;br /&gt;
:: Make a worst-case estimate and only ask for that duration&lt;br /&gt;
:: Asking for less of Mufasa&amp;#039;s time improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Debug and test code using less powerful QOSes before running it with more powerful QOSes&lt;br /&gt;
:: Your test jobs will get a higher priority and your FairShare will improve&lt;br /&gt;
&lt;br /&gt;
; Cancel jobs that you don&amp;#039;t need them anymore&lt;br /&gt;
:: Always use [[User_Jobs#Cancelling_a_job_with_scancel|scancel]] to delete completed (or crashed) jobs: your Fairshare will improve&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Protip: it&amp;#039;s a good idea to [[User Jobs#Looking for unused GPUs|check for unused GPUs]] before choosing what to request. Requesting a GPU that is currently idle will help your job get executed sooner.&lt;br /&gt;
&lt;br /&gt;
= SLURM partitions =&lt;br /&gt;
&lt;br /&gt;
Partitions are another mechanism provided by SLURM to create different levels of access to system resources. Since in Mufasa 2.0 access to resources is controlled via [[#SLURM Quality of Service (QOS)|QOSes]], partitions are not very relevant. (This is one of the main differences between Mufasa 1.0 and Mufasa 2.0.)&lt;br /&gt;
&lt;br /&gt;
Note, however, that the default values for some features of SLURM jobs (e.g., duration) are [[#Default values|set by the partition]].&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, there is a single SLURM partition, called &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt;, and all jobs run on it. The state of &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt; can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sinfo -o &amp;quot;%10P %5a %9T %11L %10l&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PARTITION  AVAIL STATE     DEFAULTTIME TIMELIMIT &lt;br /&gt;
jobs*      up    idle      1:00:00     3-00:00:00&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where columns correspond to the following information:&lt;br /&gt;
&lt;br /&gt;
:; PARTITION&lt;br /&gt;
:: name of the partition; the asterisks indicates that it&amp;#039;s the default one&lt;br /&gt;
&lt;br /&gt;
:; AVAIL&lt;br /&gt;
:: state/availability of the partition: see [[#Partition availability|below]]&lt;br /&gt;
&lt;br /&gt;
:; STATE&lt;br /&gt;
:: state (using [https://slurm.schedmd.com/sinfo.html#SECTION_NODE-STATE-CODES these codes])&lt;br /&gt;
:: typical values are &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;mixed&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that some of the resources are busy executing jobs while other are idle, and &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;allocated&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that all of the resources are in use&lt;br /&gt;
&lt;br /&gt;
:; DEFAULTTIME&lt;br /&gt;
:: default runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:; TIMELIMIT&lt;br /&gt;
:: maximum runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
The asterisk at the end of the partition name indicates the default partition, i.e. the one on which jobs which do not ask for a specific partition are run.&lt;br /&gt;
&lt;br /&gt;
Command &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; does not tell you about the &amp;#039;&amp;#039;jobs&amp;#039;&amp;#039; submitted to a partition. This information is obtained, instead, with [[User Jobs#Inspecting jobs with squeue|command &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;]].&lt;br /&gt;
&lt;br /&gt;
== Partition availability ==&lt;br /&gt;
&lt;br /&gt;
The most important information that &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; provides is the &amp;#039;&amp;#039;&amp;#039;availability&amp;#039;&amp;#039;&amp;#039; (also called &amp;#039;&amp;#039;&amp;#039;state&amp;#039;&amp;#039;&amp;#039;) of partitions. This is shown in column &amp;quot;AVAIL&amp;quot;. Possible partition states are:&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is available&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed as soon as resources allow&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is in the process of becoming unavailable (i.e., of entering the &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; state: see below)&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is unavailable&lt;br /&gt;
:: There are no running jobs&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
When a partition goes from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; no harm is done to running jobs. In a normally functioning SLURM system, the passage from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; happens only when no jobs are running on the partition. If (e.g., due to a malfunction) the passage happens with jobs still running, they get killed.&lt;br /&gt;
&lt;br /&gt;
A partition in state &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; requires intervention by a [[Roles|Job Administrator]] to be restored to &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
== Default values ==&lt;br /&gt;
&lt;br /&gt;
The features of SLURM partitions, including the &amp;#039;&amp;#039;&amp;#039;default values&amp;#039;&amp;#039;&amp;#039; which are applied to jobs that do not make explicit requests, can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scontrol show partition&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PartitionName=jobs&lt;br /&gt;
   AllowGroups=ALL AllowAccounts=ALL AllowQos=nogpu,gpulight,gpu,gpuwide,gpuheavy-20,gpuheavy-40&lt;br /&gt;
   AllocNodes=ALL Default=YES QoS=N/A&lt;br /&gt;
   DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO&lt;br /&gt;
   MaxNodes=UNLIMITED MaxTime=3-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED&lt;br /&gt;
   Nodes=gn01&lt;br /&gt;
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO&lt;br /&gt;
   OverTimeLimit=NONE PreemptMode=OFF&lt;br /&gt;
   State=UP TotalCPUs=48 TotalNodes=1 SelectTypeParameters=NONE&lt;br /&gt;
   JobDefaults=(null)&lt;br /&gt;
   DefMemPerNode=4096 MaxMemPerNode=UNLIMITED&lt;br /&gt;
   TRES=cpu=48,mem=1011435M,node=1,billing=49,gres/gpu=13,gres/gpu:3g.20gb=5,gres/gpu:40gb=3,gres/gpu:4g.20gb=5&lt;br /&gt;
   TRESBillingWeights=cpu=1.0,gres/gpu:3g.20gb=6.0,gres/gpu:4g.20gb=6.0,gres/gpu:40gb=6.0,mem=0.05g&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In the output of &amp;lt;code&amp;gt;scontrol show partition&amp;lt;/code&amp;gt; the most relevant default values for Mufasa users are:&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefaultTime&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default execution time assigned to a job run on the partition (e.g., 1 hour)&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefMemPerNode&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default amount of RAM assigned to a job run on the partition (e.g., 4GB)&lt;br /&gt;
&lt;br /&gt;
= System resources subjected to limitations =&lt;br /&gt;
&lt;br /&gt;
In systems based on SLURM like Mufasa, &amp;#039;&amp;#039;&amp;#039;TRES (Trackable RESources)&amp;#039;&amp;#039;&amp;#039; are (from [https://slurm.schedmd.com/tres.html SLURM&amp;#039;s documentation] &amp;quot;&amp;#039;&amp;#039;resources that can be tracked for usage or used to enforce limits against.&amp;#039;&amp;#039;&amp;quot;&lt;br /&gt;
&lt;br /&gt;
TRES include CPUs, RAM and &amp;#039;&amp;#039;&amp;#039;GRES&amp;#039;&amp;#039;&amp;#039;. The last term stands for &amp;#039;&amp;#039;Generic RESources&amp;#039;&amp;#039; that a job may need for its execution. In Mufasa, the only &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; resources are the GPUs.&lt;br /&gt;
&lt;br /&gt;
== &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax ==&lt;br /&gt;
&lt;br /&gt;
To ask SLURM to assign GRES resources (i.e., GPUs) to a job, a special syntax must be used. Precisely, the name of each GPU resource takes the form&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:Name:Type&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
Considering the [[System#CPUs and GPUs|GPU complement of Mufasa]], &amp;lt;code&amp;gt;Type&amp;lt;/code&amp;gt; takes the following values:&lt;br /&gt;
&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:40gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 40 Gbytes of RAM&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:4g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 4 compute units&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 3 compute units&lt;br /&gt;
&lt;br /&gt;
So, for instance,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
identifies a resource corresponding to a GPU with 20 GB of RAM and 3 compute units.&lt;br /&gt;
&lt;br /&gt;
When asking for a GRES resource (e.g., in an &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command or an &amp;lt;code&amp;gt;SBATCH&amp;lt;/code&amp;gt; directive of an [[User Jobs#Using execution scripts to run jobs|execution script]]), the syntax required by SLURM is&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:&amp;lt;Type&amp;gt;:&amp;lt;Quantity&amp;gt;&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
where &amp;lt;code&amp;gt;Quantity&amp;lt;/code&amp;gt; is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of type &amp;lt;code&amp;gt;4g.20gb&amp;lt;/code&amp;gt; the syntax is&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:4g.20gb:2&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SLURM&amp;#039;s &amp;#039;&amp;#039;generic resources&amp;#039;&amp;#039; are defined in &amp;lt;code&amp;gt;/etc/slurm/gres.conf&amp;lt;/code&amp;gt;. In order to make GPUs available to SLURM&amp;#039;s &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; management, Mufasa makes use of Nvidia&amp;#039;s [https://developer.nvidia.com/nvidia-management-library-nvml NVML library]. For additional information see [https://slurm.schedmd.com/gres.html SLURM&amp;#039;s documentation].&lt;/div&gt;</summary>
		<author><name>GiulioFontana</name></author>
	</entry>
	<entry>
		<id>https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2594</id>
		<title>SLURM</title>
		<link rel="alternate" type="text/html" href="https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2594"/>
		<updated>2026-11-18T09:01:28Z</updated>

		<summary type="html">&lt;p&gt;GiulioFontana: /* Elements determining job priority */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page presents the features of SLURM that are most relevant to Mufasa&amp;#039;s [[Roles|Job Users]]. Job Users can submit jobs for execution, cancel their own jobs, and see other users&amp;#039; jobs (but not intervene on them).&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa &amp;#039;&amp;#039;&amp;#039;must use SLURM&amp;#039;&amp;#039;&amp;#039; to run resource-heavy processes, i.e. computing jobs that require one or more of the following:&lt;br /&gt;
* GPUs&lt;br /&gt;
* multiple CPUs&lt;br /&gt;
* powerful CPUs&lt;br /&gt;
* a significant amount of RAM&lt;br /&gt;
&lt;br /&gt;
In fact, only processes run via SLURM have access to all the resources of Mufasa. Processes run outside SLURM are executed by the [[System#Login server|login server]] virtual machine, which has minimal resources and no GPUs. Using SLURM is therefore the only way to execute resource-heavy jobs on Mufasa (this is a key difference between Mufasa 1.0 and Mufasa 2.0).&lt;br /&gt;
&lt;br /&gt;
= SLURM in a nutshell =&lt;br /&gt;
&lt;br /&gt;
Computation jobs on Mufasa needs to be launched via [[System#The SLURM job scheduling system|SLURM]]. SLURM provides jobs with access to the [[#System resources subjected to limitations|physical resources]] of Mufasa, such as CPUs, GPUs and RAM. Thanks to SLURM, processing jobs share system resources, optimising their occupation and availability. &lt;br /&gt;
&lt;br /&gt;
When a user runs a job, the job does not get executed immediately and is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039;. SLURM executes jobs according to their order in the queue: the top job in the queue gets executed as soon as the necessary resources are available, while jobs lower in the queue wait longer. The position of a job in the queue is due to the &amp;#039;&amp;#039;&amp;#039;[[#Job priority|priority]]&amp;#039;&amp;#039;&amp;#039; assigned to it by SLURM, with higher-priority jobs closer to the top. As a general rule:&lt;br /&gt;
&lt;br /&gt;
;: &amp;#039;&amp;#039;&amp;#039;the greater the fraction of Mufasa&amp;#039;s overall resources that a job asks for, the lower the job&amp;#039;s priority will be&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
The priority mechanism is used to encourage users to use Mufasa&amp;#039;s resources (i.e.: GPUs, CPUs, RAM, execution time) in an effective and equitable manner. This page includes a [[#How_to_maximise_the_priority_of_your_jobs|chart explaining how to maximise the priority of your jobs]].&lt;br /&gt;
&lt;br /&gt;
The &amp;#039;&amp;#039;&amp;#039;time&amp;#039;&amp;#039;&amp;#039; available to a job for its execution is controlled by SLURM. When a user requests execution of a job, they must specify the duration of the time slot that the job needs. The job must complete its execution before the end of the requested time slot, because at the end of such slot it gets killed by SLURM.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0 access to system resources is managed via SLURM&amp;#039;s [[#SLURM Quality of Service (QOS)|Quality of Service (QOS)]] mechanism (Mufasa 1.0 used [[#SLURM_partitions|partitions]] instead). While [[User Jobs#Running jobs with SLURM|launching a processing job via SLURM]], the user must always specify the chosen QOS. QOSes differ in the set of resources that they provide access to because each of them is designed to fit a given type of job.&lt;br /&gt;
&lt;br /&gt;
Mufasa sets limits to the number of jobs by the same user. This page includes a [[#Limits on jobs by the same user|table summarising such limits]].&lt;br /&gt;
&lt;br /&gt;
= SLURM Quality of Service (QOS) =&lt;br /&gt;
&lt;br /&gt;
Through &amp;#039;&amp;#039;&amp;#039;Quality of Services&amp;#039;&amp;#039;&amp;#039; (&amp;#039;&amp;#039;&amp;#039;QOSes&amp;#039;&amp;#039;&amp;#039;), SLURM lets system configurators assign a name to a set of related constraints.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, QOSes are used to define different levels of access to the server&amp;#039;s resources. When [[User Jobs|executing a job with SLURM]], a user must always &amp;#039;&amp;#039;&amp;#039;specify the QOS&amp;#039;&amp;#039;&amp;#039; that their job will use: this choice, in turn, determines what resources the job is able to access and influences the [[#Job priority|priority]] of the job.&lt;br /&gt;
&lt;br /&gt;
Mufasa&amp;#039;s QOSes and their features can be inspected with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list qos format=name%-11,priority,MaxSubmit,maxwall,maxtres%-80,mintres%-18&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following (see below for the same data in table form):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
Name          Priority MaxSubmit     MaxWall MaxTRES                                                                          MinTRES            &lt;br /&gt;
----------- ---------- --------- ----------- -------------------------------------------------------------------------------- ------------------ &lt;br /&gt;
normal               0                                                                                                                           &lt;br /&gt;
nogpu                4         1  3-00:00:00 cpu=16,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=128G                    &lt;br /&gt;
gpuheavy-20          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=2,mem=128G             gres/gpu:4g.20gb=1 &lt;br /&gt;
gpuheavy-40          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=1,gres/gpu:4g.20gb=0,mem=128G             gres/gpu:40gb=1    &lt;br /&gt;
gpulight             8         1    12:00:00 cpu=2,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpu                  2         1  1-00:00:00 cpu=8,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpuwide              2         2  1-00:00:00 cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=1,mem=64G              gres/gpu:4g.20gb=1 &lt;br /&gt;
build               32         1    02:00:00 cpu=2,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=16G&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The columns of this output are the following:&lt;br /&gt;
&lt;br /&gt;
:; Name&lt;br /&gt;
:: name of the QOS&lt;br /&gt;
&lt;br /&gt;
:; Priority&lt;br /&gt;
:: priority tier associated to the QOS (higher value = higher priority): see [[#Job priority|Job priority]] for details&lt;br /&gt;
&lt;br /&gt;
:; MaxSubmit&lt;br /&gt;
:: maximum number of jobs from a single user that can be submitted to SLURM with this QOS; submitted jobs include both running and queued jobs&lt;br /&gt;
:: See [[#Limits on jobs by the same user|Limits on jobs by the same user]] for an overview of the limits on jobs set by Mufasa.&lt;br /&gt;
&lt;br /&gt;
:; MaxWall&lt;br /&gt;
:: maximum wall clock duration of the jobs using the QOS (after which they are killed by SLURM), in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
:: For some QOSes these are not set: it means that they are determined by the [[#SLURM partitions|partition]]. Partitions also define the [[#Default values|default duration]] of jobs.&lt;br /&gt;
&lt;br /&gt;
:; MaxTRES&lt;br /&gt;
:: amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) available to a job using the QOS, where&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;cpu=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of CPUs (i.e., processor cores) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of CPUs specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;gres/&amp;#039;&amp;#039;gpu:Type&amp;#039;&amp;#039;=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of GPUs of class &amp;lt;code&amp;gt;&amp;#039;&amp;#039;Type&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; (see [[User Jobs#gres syntax|&amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax]]) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; (for QOSes that allow access to GPUs) if not specified, the job cannot be launched&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;mem=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;G&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum amount of system RAM is &amp;#039;&amp;#039;K&amp;#039;&amp;#039; GBytes&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of RAM specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
&lt;br /&gt;
:; MinTRES&lt;br /&gt;
:: minimum amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) that a job using the QOS must request in order to actually get executed by SLURM.&lt;br /&gt;
:: (If your job does not actually need these resources, you&amp;#039;ve chosen the wrong QOS: you can use one with a higher priority.)&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; QOS is the default one, and exists only to ensure that users always specify a QOS when running a job. Since &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; has zero priority and no resources, a job run using this QOS would never be run.&lt;br /&gt;
&lt;br /&gt;
The information provided by the &amp;lt;code&amp;gt;sacctmgr list qos&amp;lt;/code&amp;gt; command above is summarised by the following table:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! QOS&lt;br /&gt;
! Priority&amp;lt;/br&amp;gt;tier&lt;br /&gt;
! Max&amp;lt;/br&amp;gt;Submit&lt;br /&gt;
! MaxWall&amp;lt;/br&amp;gt;[h]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;CPUs&lt;br /&gt;
! max RAM&amp;lt;/br&amp;gt;[GB]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;3g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;4g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;40GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | build&lt;br /&gt;
| 32&lt;br /&gt;
| 1&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 16&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpulight&lt;br /&gt;
| 8&lt;br /&gt;
| 1&lt;br /&gt;
| 12&lt;br /&gt;
| 2&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | nogpu&lt;br /&gt;
| 4&lt;br /&gt;
| 1&lt;br /&gt;
| 72&lt;br /&gt;
| 16&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpu&lt;br /&gt;
| 2&lt;br /&gt;
| 1&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuwide&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-20&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| 2 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-40&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
A key piece of information in the table above is the &amp;#039;&amp;#039;&amp;#039;priority tier&amp;#039;&amp;#039;&amp;#039; associated to each QOS. In Mufasa 2.0, priority tiers are used to encourage users to use the &amp;#039;&amp;#039;&amp;#039;least powerful QOS that is compatible with their needs&amp;#039;&amp;#039;&amp;#039;, where &amp;quot;powerful&amp;quot; means &amp;quot;rich with resources&amp;quot;. Less powerful QOSes increase the priority of the jobs that use them, so these jobs tend to be executed sooner. See [[#Job priority|Job priority]] for details about how priority affects the execution order of jobs in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
== The &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS ==&lt;br /&gt;
&lt;br /&gt;
This QOS is specifically designed to be used by Mufasa users to &amp;#039;&amp;#039;&amp;#039;quickly build [[System#Containers|container images]]&amp;#039;&amp;#039;&amp;#039;. Its associated priority tier is very high: this, combined with the fact that jobs using this QOS require few resources, means that such jobs usually get executed very soon.&lt;br /&gt;
&lt;br /&gt;
On the other hand, the limited resources and the lack of access to the GPUs make the &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS unsuitable for other tasks.&lt;br /&gt;
&lt;br /&gt;
See [[Singularity#Building Singularity images|Building Singularity images]] for directions about building Singularity container images.&lt;br /&gt;
&lt;br /&gt;
== Restricted QOSes ==&lt;br /&gt;
&lt;br /&gt;
In Mufasa, the most powerful QOSes are reserved to researchers (called &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users: these include academic personnel and Ph.D. students). M.Sc. students, called &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users, cannot use them while running jobs.&lt;br /&gt;
&lt;br /&gt;
See [[#research users and students users|below]] to understand how &amp;#039;&amp;#039;user categories&amp;#039;&amp;#039; work in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
= &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users =&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa belong to two &amp;#039;&amp;#039;&amp;#039;user categories&amp;#039;&amp;#039;&amp;#039;. Categories provide users with different access to  to Mufasa&amp;#039;s resources: the idea being to provide researchers with more access while still letting students benefit from the server&amp;#039;s capabilities.&lt;br /&gt;
&lt;br /&gt;
User categories in Mufasa are the following:&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. academic personnel and Ph.D. students&lt;br /&gt;
::: * can use all QOSes, including the the [[#Restricted QOSes|restricted ones]]&lt;br /&gt;
::: * their jobs have a higher &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a higher number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. M.Sc. students&lt;br /&gt;
::: * cannot use the [[#Restricted QOSes|restricted QOSes]]&lt;br /&gt;
::: * their jobs have a lower &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a lower number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
You can inspect the differences in priority and running jobs between &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association format=&amp;quot;account,priority,maxjobs&amp;quot; | grep -E &amp;#039;Account|research|students&amp;#039;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
   Account   Priority MaxJobs &lt;br /&gt;
  research          4       2 &lt;br /&gt;
  students          1       1&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
To know what limits apply to your own user, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association where user=$USER format=&amp;quot;user,priority,maxjobs,qos%-60&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
      User   Priority MaxJobs QOS                                                          &lt;br /&gt;
---------- ---------- ------- ------------------------------------------------------------ &lt;br /&gt;
    preali          4       2 build,gpu,gpuheavy-20,gpuheavy-40,gpulight,gpuwide,nogpu&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The list under &amp;quot;QOS&amp;quot; shows what QOSes your user is allowed to use when [[User Jobs|running jobs]]. &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users can use all of them, while &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users can only access a subset of them.&lt;br /&gt;
&lt;br /&gt;
= Limits on jobs by the same user =&lt;br /&gt;
&lt;br /&gt;
Mufasa enforces limits on the number of jobs from a single user. Such limits aim at preventing users from &amp;quot;hogging&amp;quot; system resources, and apply to:&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;submitted jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that the user asked SLURM to execute, each of which may currently be either running or queued&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;running jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that are currently in execution&lt;br /&gt;
&lt;br /&gt;
Limits on submitted jobs are set via [[#SLURM Quality of Service (QOS)|QOSes]], while limits on running jobs are set via [[#research users and students users|user category]].&lt;br /&gt;
&lt;br /&gt;
The following table summarises what limits exist:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! &lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;running&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;submitted&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | global limits&amp;lt;br/&amp;gt;(system-wide)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to limits set by individual QOSes (below)&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | limits for&amp;lt;br/&amp;gt;individual&amp;lt;/br&amp;gt;QOSes&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to global limits (above)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for the&amp;#039;&amp;#039; &amp;lt;code&amp;gt;gpuwide&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;QOS&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for each of the other QOSes&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
= Job priority =&lt;br /&gt;
&lt;br /&gt;
Once the execution of a job has been requested, the job is not run immediately: it is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039; by SLURM, together with all the other jobs awaiting execution. The job on top of the queue at any time is the first to be put into execution as soon as the resources it requires are available. &lt;br /&gt;
&lt;br /&gt;
The order of the items in the job queue depends on their &amp;#039;&amp;#039;&amp;#039;priority&amp;#039;&amp;#039;&amp;#039;. The job with the highest priority is put by SLURM on top of the queue, while all other jobs are put in the queue in order of descending priority.&lt;br /&gt;
&lt;br /&gt;
The goal of SLURM is to maximise resource availability: i.e., to ensure the shorter possible wait time before job execution. To achieve this goal, in Mufasa SLURM is configured to &amp;#039;&amp;#039;&amp;#039;encourage users not to ask for resources or execution time that their job doesn&amp;#039;t need&amp;#039;&amp;#039;&amp;#039;. This is done via the priority mechanism: the more resources and/or time a job requests, the lower its priority will be; and the later it will be executed.&lt;br /&gt;
&lt;br /&gt;
Priority management in Mufasa is designed to set up a &amp;#039;&amp;#039;&amp;#039;virtuous cycle&amp;#039;&amp;#039;&amp;#039; where users, by carefully choosing what to ask for, obtain two results:&lt;br /&gt;
* they ensure that their job is executed as soon as possible;&lt;br /&gt;
* they leave as much as possible of Mufasa&amp;#039;s resources free for other users&amp;#039;s jobs.&lt;br /&gt;
&lt;br /&gt;
== Elements determining job priority ==&lt;br /&gt;
In Mufasa, the priority of a job is computed by SLURM according to the following elements:&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#research users and students users|User category]]&amp;#039;&amp;#039;&amp;#039; (i.e., &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;)&lt;br /&gt;
::: Used to provide higher priority to jobs run by &amp;#039;&amp;#039;&amp;#039;research personnel&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#SLURM Quality of Service (QOS)|QOS]]&amp;#039;&amp;#039;&amp;#039; used by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;less resources&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Number of CPUs&amp;#039;&amp;#039;&amp;#039; requested by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;a lower number of CPUs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job Age&amp;#039;&amp;#039;&amp;#039;, i.e. the time that the job has been waiting in the queue&lt;br /&gt;
::: Used to provide higher priority to jobs which have been &amp;#039;&amp;#039;&amp;#039;queued for a longer time&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
::: Note: age is computed only when a queued job is &amp;#039;&amp;#039;eligible for running&amp;#039;&amp;#039;. Time when a job cannot run because of the limits on the number of running jobs by the same user does not count.&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;FairShare&amp;#039;&amp;#039;&amp;#039;, i.e. a factor computed by SLURM to balance use of the system by different users&lt;br /&gt;
::: Used to provide higher priority to jobs by users who &amp;#039;&amp;#039;&amp;#039;used Mufasa&amp;#039;s resources less than others&amp;#039;&amp;#039;&amp;#039; (see [[#How FairShare works|below]])&lt;br /&gt;
&lt;br /&gt;
You can understand how each of these elements contributes to the priority of the jobs waiting for execution (both yours and of other users) with command &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;. A tailored version of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; that only displays the parameters listed above is the following:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sprio -o &amp;quot;%6i %9u %.8Y %.10B %.5Q %.9T %.5A %.5F&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Here is an example of the output of the command above:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
JOBID  USER      PRIORITY      ASSOC   QOS      TRES   AGE FAIRS&lt;br /&gt;
5790   amoglia       3495       2500   375  cpu=-800   933   487&lt;br /&gt;
5791   qliu          3010       2500   375  cpu=-800   916    19&lt;br /&gt;
5817   mzoccola      2336        625   750  cpu=-800   495  1266&lt;br /&gt;
5822   fbottazzi     2235        625   750  cpu=-600   467   994&lt;br /&gt;
5837   lpassoni      2508       2500   750  cpu=-800     0    58&lt;br /&gt;
5838   bagostini     4437        625  3000  cpu=-200     0  1013&lt;br /&gt;
5839   flazzaro      1549        625   750  cpu=-800     0   974&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
In it, column PRIORITY provides the overall priority of each job. Each priority value is the sum of the five values to its right; in particular, column ASSOC corresponds to user category, while column TRES corresponds to the number of CPUs. Note that the contribution of TRES to priority is &amp;#039;&amp;#039;negative&amp;#039;&amp;#039;, i.e. it is a penalty; its value is proportional to the number of CPUs requested by the job.&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;Things to consider when using &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
* Its output is only relevant for jobs &amp;#039;&amp;#039;not yet in execution&amp;#039;&amp;#039;. For an already running job, the values shown by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; correspond to the priority that the job would get if evaluated now, which generally differs from its actual priority.&lt;br /&gt;
* Jobs listed by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; may temporarily differ from jobs listed by &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;: this is due to the fact that SLURM updates its internal data with a periodicity of a few minutes.&lt;br /&gt;
&lt;br /&gt;
=== How FairShare works ===&lt;br /&gt;
In Mufasa 2.0, the FairShare of each user is computed according to these rules:&lt;br /&gt;
# It considers the quantity used by (current and past) user jobs of:&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;CPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of CPUs used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;RAM&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the amount of RAM used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;GPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of GPUs used&lt;br /&gt;
#** note: 40 GB GPUs have double the impact on FairShare than 20 GB GPUs&lt;br /&gt;
# The impact on FairShare of any use of resources is &amp;#039;&amp;#039;&amp;#039;proportional to its duration&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
#* example: using 32 GB of RAM for 48h has the same impact of using 64 GB of RAM for 24h&lt;br /&gt;
# FairShare has a &amp;quot;fading memory&amp;quot;: i.e., resource use has less and less impact on FairShare the farther in the past it is&lt;br /&gt;
#* consequence: over time, the &amp;quot;history&amp;quot; of a user gets forgotten by FairShare&lt;br /&gt;
&lt;br /&gt;
== How to maximise the priority of your jobs ==&lt;br /&gt;
&lt;br /&gt;
Every time you run a SLURM job, follow these guidelines:&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Choose the less powerful QOS compatible with the needs of your job&lt;br /&gt;
:: QOSes with access to less resources have higher base priority&lt;br /&gt;
&lt;br /&gt;
; Only request CPUs and RAM that your job really needs&lt;br /&gt;
:: Usually, code exploits multiple CPUs only if specifically designed to do so: if unsure, only ask for 1 CPU&lt;br /&gt;
:: The fewer resources your job asks for, the less it will wait before they become available&lt;br /&gt;
:: The more CPUs your job asks for, the stronger the penalty to its priority&lt;br /&gt;
:: Asking for fewer CPUs and/or less RAM improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Do not request more time than your jobs needs to complete&lt;br /&gt;
:: Make a worst-case estimate and only ask for that duration&lt;br /&gt;
:: Asking for less of Mufasa&amp;#039;s time improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Debug and test code using less powerful QOSes before running it with more powerful QOSes&lt;br /&gt;
:: Your test jobs will get a higher priority and your FairShare will improve&lt;br /&gt;
&lt;br /&gt;
; Cancel jobs that you don&amp;#039;t need them anymore&lt;br /&gt;
:: Always use [[User_Jobs#Cancelling_a_job_with_scancel|scancel]] to delete completed (or crashed) jobs: your Fairshare will improve&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Protip: it&amp;#039;s a good idea to [[User Jobs#Looking for unused GPUs|check for unused GPUs]] before choosing what to request. Requesting a GPU that is currently idle will help your job get executed sooner.&lt;br /&gt;
&lt;br /&gt;
= SLURM partitions =&lt;br /&gt;
&lt;br /&gt;
Partitions are another mechanism provided by SLURM to create different levels of access to system resources. Since in Mufasa 2.0 access to resources is controlled via [[#SLURM Quality of Service (QOS)|QOSes]], partitions are not very relevant. (This is one of the main differences between Mufasa 1.0 and Mufasa 2.0.)&lt;br /&gt;
&lt;br /&gt;
Note, however, that the default values for some features of SLURM jobs (e.g., duration) are [[#Default values|set by the partition]].&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, there is a single SLURM partition, called &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt;, and all jobs run on it. The state of &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt; can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sinfo -o &amp;quot;%10P %5a %9T %11L %10l&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PARTITION  AVAIL STATE     DEFAULTTIME TIMELIMIT &lt;br /&gt;
jobs*      up    idle      1:00:00     3-00:00:00&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where columns correspond to the following information:&lt;br /&gt;
&lt;br /&gt;
:; PARTITION&lt;br /&gt;
:: name of the partition; the asterisks indicates that it&amp;#039;s the default one&lt;br /&gt;
&lt;br /&gt;
:; AVAIL&lt;br /&gt;
:: state/availability of the partition: see [[#Partition availability|below]]&lt;br /&gt;
&lt;br /&gt;
:; STATE&lt;br /&gt;
:: state (using [https://slurm.schedmd.com/sinfo.html#SECTION_NODE-STATE-CODES these codes])&lt;br /&gt;
:: typical values are &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;mixed&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that some of the resources are busy executing jobs while other are idle, and &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;allocated&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that all of the resources are in use&lt;br /&gt;
&lt;br /&gt;
:; DEFAULTTIME&lt;br /&gt;
:: default runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:; TIMELIMIT&lt;br /&gt;
:: maximum runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
The asterisk at the end of the partition name indicates the default partition, i.e. the one on which jobs which do not ask for a specific partition are run.&lt;br /&gt;
&lt;br /&gt;
Command &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; does not tell you about the &amp;#039;&amp;#039;jobs&amp;#039;&amp;#039; submitted to a partition. This information is obtained, instead, with [[User Jobs#Inspecting jobs with squeue|command &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;]].&lt;br /&gt;
&lt;br /&gt;
== Partition availability ==&lt;br /&gt;
&lt;br /&gt;
The most important information that &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; provides is the &amp;#039;&amp;#039;&amp;#039;availability&amp;#039;&amp;#039;&amp;#039; (also called &amp;#039;&amp;#039;&amp;#039;state&amp;#039;&amp;#039;&amp;#039;) of partitions. This is shown in column &amp;quot;AVAIL&amp;quot;. Possible partition states are:&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is available&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed as soon as resources allow&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is in the process of becoming unavailable (i.e., of entering the &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; state: see below)&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is unavailable&lt;br /&gt;
:: There are no running jobs&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
When a partition goes from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; no harm is done to running jobs. In a normally functioning SLURM system, the passage from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; happens only when no jobs are running on the partition. If (e.g., due to a malfunction) the passage happens with jobs still running, they get killed.&lt;br /&gt;
&lt;br /&gt;
A partition in state &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; requires intervention by a [[Roles|Job Administrator]] to be restored to &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
== Default values ==&lt;br /&gt;
&lt;br /&gt;
The features of SLURM partitions, including the &amp;#039;&amp;#039;&amp;#039;default values&amp;#039;&amp;#039;&amp;#039; which are applied to jobs that do not make explicit requests, can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scontrol show partition&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PartitionName=jobs&lt;br /&gt;
   AllowGroups=ALL AllowAccounts=ALL AllowQos=nogpu,gpulight,gpu,gpuwide,gpuheavy-20,gpuheavy-40&lt;br /&gt;
   AllocNodes=ALL Default=YES QoS=N/A&lt;br /&gt;
   DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO&lt;br /&gt;
   MaxNodes=UNLIMITED MaxTime=3-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED&lt;br /&gt;
   Nodes=gn01&lt;br /&gt;
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO&lt;br /&gt;
   OverTimeLimit=NONE PreemptMode=OFF&lt;br /&gt;
   State=UP TotalCPUs=48 TotalNodes=1 SelectTypeParameters=NONE&lt;br /&gt;
   JobDefaults=(null)&lt;br /&gt;
   DefMemPerNode=4096 MaxMemPerNode=UNLIMITED&lt;br /&gt;
   TRES=cpu=48,mem=1011435M,node=1,billing=49,gres/gpu=13,gres/gpu:3g.20gb=5,gres/gpu:40gb=3,gres/gpu:4g.20gb=5&lt;br /&gt;
   TRESBillingWeights=cpu=1.0,gres/gpu:3g.20gb=6.0,gres/gpu:4g.20gb=6.0,gres/gpu:40gb=6.0,mem=0.05g&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In the output of &amp;lt;code&amp;gt;scontrol show partition&amp;lt;/code&amp;gt; the most relevant default values for Mufasa users are:&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefaultTime&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default execution time assigned to a job run on the partition (e.g., 1 hour)&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefMemPerNode&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default amount of RAM assigned to a job run on the partition (e.g., 4GB)&lt;br /&gt;
&lt;br /&gt;
= System resources subjected to limitations =&lt;br /&gt;
&lt;br /&gt;
In systems based on SLURM like Mufasa, &amp;#039;&amp;#039;&amp;#039;TRES (Trackable RESources)&amp;#039;&amp;#039;&amp;#039; are (from [https://slurm.schedmd.com/tres.html SLURM&amp;#039;s documentation] &amp;quot;&amp;#039;&amp;#039;resources that can be tracked for usage or used to enforce limits against.&amp;#039;&amp;#039;&amp;quot;&lt;br /&gt;
&lt;br /&gt;
TRES include CPUs, RAM and &amp;#039;&amp;#039;&amp;#039;GRES&amp;#039;&amp;#039;&amp;#039;. The last term stands for &amp;#039;&amp;#039;Generic RESources&amp;#039;&amp;#039; that a job may need for its execution. In Mufasa, the only &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; resources are the GPUs.&lt;br /&gt;
&lt;br /&gt;
== &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax ==&lt;br /&gt;
&lt;br /&gt;
To ask SLURM to assign GRES resources (i.e., GPUs) to a job, a special syntax must be used. Precisely, the name of each GPU resource takes the form&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:Name:Type&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
Considering the [[System#CPUs and GPUs|GPU complement of Mufasa]], &amp;lt;code&amp;gt;Type&amp;lt;/code&amp;gt; takes the following values:&lt;br /&gt;
&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:40gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 40 Gbytes of RAM&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:4g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 4 compute units&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 3 compute units&lt;br /&gt;
&lt;br /&gt;
So, for instance,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
identifies a resource corresponding to a GPU with 20 GB of RAM and 3 compute units.&lt;br /&gt;
&lt;br /&gt;
When asking for a GRES resource (e.g., in an &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command or an &amp;lt;code&amp;gt;SBATCH&amp;lt;/code&amp;gt; directive of an [[User Jobs#Using execution scripts to run jobs|execution script]]), the syntax required by SLURM is&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:&amp;lt;Type&amp;gt;:&amp;lt;Quantity&amp;gt;&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
where &amp;lt;code&amp;gt;Quantity&amp;lt;/code&amp;gt; is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of type &amp;lt;code&amp;gt;4g.20gb&amp;lt;/code&amp;gt; the syntax is&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:4g.20gb:2&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SLURM&amp;#039;s &amp;#039;&amp;#039;generic resources&amp;#039;&amp;#039; are defined in &amp;lt;code&amp;gt;/etc/slurm/gres.conf&amp;lt;/code&amp;gt;. In order to make GPUs available to SLURM&amp;#039;s &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; management, Mufasa makes use of Nvidia&amp;#039;s [https://developer.nvidia.com/nvidia-management-library-nvml NVML library]. For additional information see [https://slurm.schedmd.com/gres.html SLURM&amp;#039;s documentation].&lt;/div&gt;</summary>
		<author><name>GiulioFontana</name></author>
	</entry>
	<entry>
		<id>https://biohpc.deib.polimi.it/index.php?title=User_Jobs&amp;diff=2593</id>
		<title>User Jobs</title>
		<link rel="alternate" type="text/html" href="https://biohpc.deib.polimi.it/index.php?title=User_Jobs&amp;diff=2593"/>
		<updated>2026-11-04T14:33:26Z</updated>

		<summary type="html">&lt;p&gt;GiulioFontana: /* Options of srun and sbatch */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;= Running jobs with SLURM =&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa &amp;#039;&amp;#039;&amp;#039;must&amp;#039;&amp;#039;&amp;#039; use SLURM to run resource-heavy processes, i.e. computing jobs that require one or more of the following:&lt;br /&gt;
* GPUs&lt;br /&gt;
* multiple CPUs&lt;br /&gt;
* powerful CPUs&lt;br /&gt;
* a significant amount of RAM.&lt;br /&gt;
&lt;br /&gt;
In fact, only processes run via SLURM have access to all the resources of Mufasa. Processes run outside SLURM are executed by the [[System#Login server|login server]] virtual machine, which has minimal resources and no GPUs. Using SLURM is therefore the only way to execute resource-heavy jobs on Mufasa. This is a key difference between Mufasa 1.0 and [[System#Mufasa 2.0|Mufasa 2.0]].&lt;br /&gt;
&lt;br /&gt;
== &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;sbatch&amp;lt;/code&amp;gt; ==&lt;br /&gt;
&lt;br /&gt;
SLURM provides two commands to run jobs, called [https://slurm.schedmd.com/srun.html srun] and [https://slurm.schedmd.com/sbatch.html sbatch]:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
srun [options] &amp;lt;command_to_be_run_via_SLURM&amp;gt;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sbatch [options] &amp;lt;command_to_be_run_via_SLURM&amp;gt;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In both cases, &amp;lt;code&amp;gt;&amp;lt;command_to_be_run_via_SLURM&amp;gt;&amp;lt;/code&amp;gt; can be any Linux program (including shell scripts). By using &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;sbatch&amp;lt;/code&amp;gt;, the  command or script specified by &amp;lt;code&amp;gt;&amp;lt;command_to_be_run_via_SLURM&amp;gt;&amp;lt;/code&amp;gt; (including any programs launched by it) are added to SLURM&amp;#039;s execution queues.&lt;br /&gt;
&lt;br /&gt;
The main difference between &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;sbatch&amp;lt;/code&amp;gt; is that the first locks the shell from which it has been launched, so it is only really suitable for &amp;#039;&amp;#039;&amp;#039;interactive jobs&amp;#039;&amp;#039;&amp;#039;: i.e., processes that use the console to interact with their user during job execution. &amp;lt;code&amp;gt;sbatch&amp;lt;/code&amp;gt;, on the other side, does not lock the shell and simply adds the job to the queue, but does not allow the user to interact with the process while it is running.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;sbatch&amp;lt;/code&amp;gt; provides an additional possibility: &amp;lt;code&amp;gt;&amp;lt;command_to_be_run_via_SLURM&amp;gt;&amp;lt;/code&amp;gt; can in fact be an [[#Using execution scripts to run jobs|&amp;#039;&amp;#039;&amp;#039;execution script&amp;#039;&amp;#039;&amp;#039;]], i.e. a special (and SLURM-specific) type of Linux shell script that includes &amp;#039;&amp;#039;&amp;#039;SBATCH directives&amp;#039;&amp;#039;&amp;#039;. SBATCH directives can be used to specify the values of some of the parameters that would otherwise have to be set using the &amp;lt;code&amp;gt;[options]&amp;lt;/code&amp;gt; part of the &amp;lt;code&amp;gt;sbatch&amp;lt;/code&amp;gt; command. This is handy because it allows to write down the parameters in an execution script instead of having to write them in the command line while launching a job, which greatly reduces the possibility of mistakes. Also, an execution script is easy to keep and reuse.&lt;br /&gt;
&lt;br /&gt;
Immediately after a &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;sbatch&amp;lt;/code&amp;gt; command is launched by a user, SLURM outputs a message informing the user that the job has been queued. The output is similar to this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
srun: job 10849 queued and waiting for resources&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The shell is now locked while SLURM prepares the execution of the user program ([[#Detaching from a running job with screen|if you are using &amp;lt;code&amp;gt;screen&amp;lt;/code&amp;gt; you can detach from that shell and come back later]]). &lt;br /&gt;
&lt;br /&gt;
When SLURM is ready to run the program, it prints a message similar to&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
srun: job 10849 has been allocated resources&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
and then executes the program.&lt;br /&gt;
&lt;br /&gt;
=== Options of &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;sbatch&amp;lt;/code&amp;gt; ===&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;[options]&amp;lt;/code&amp;gt; part of &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;sbatch&amp;lt;/code&amp;gt; commands is used to tell SLURM what resources the job needs to be executed the job and how much time it will need to complete its execution.&lt;br /&gt;
&lt;br /&gt;
For what concerns resources, the most important option is &amp;lt;code&amp;gt;--qos &amp;lt;qos_name&amp;gt;&amp;lt;/code&amp;gt;, specifying which SLURM [[#SLURM Quality of Service (QOS)|SLURM QOS]] the job will use. A job run with a given QOS has access to all and only the resources available to that QOS. As a consequence, all options that define how many resources to assign the job will only be able to provide the job with resources that are available to the chosen QOS. Jobs that require resources that are not available to the chosen QOS do not get executed. &lt;br /&gt;
&lt;br /&gt;
If the user forgets to use option &amp;lt;code&amp;gt;--qos &amp;lt;qos_name&amp;gt;&amp;lt;/code&amp;gt;, the job is run on the &amp;#039;&amp;#039;default qos&amp;#039;&amp;#039; (&amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt;) which has access to &amp;#039;&amp;#039;zero&amp;#039;&amp;#039; resources. Therefore it is always necessary to specify option &amp;lt;code&amp;gt;--qos &amp;lt;qos_name&amp;gt;&amp;lt;/code&amp;gt; when launching a SLURM job on Mufasa.&lt;br /&gt;
&lt;br /&gt;
More generally, the most relevant among the &amp;lt;code&amp;gt;[options]&amp;lt;/code&amp;gt; are:&lt;br /&gt;
&lt;br /&gt;
:;‑-qos=&amp;lt;qos__name&amp;gt;&lt;br /&gt;
:: specifies the [[SLURM#SLURM Quality of Service (QOS)|SLURM QOS]] that the job will use. It is mandatory to specify one.&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;Important! The chosen QOS limits the resources that can be requested, since it is not allowed to request resources (type or quantity) that exceed what is available to the chosen QOS.&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;Important! If &amp;lt;code&amp;gt;‑‑qos &amp;lt;qos_name&amp;gt;&amp;lt;/code&amp;gt; is used and options that specify how many resources to assign to the job (such as &amp;lt;code&amp;gt;‑‑mem=&amp;lt;mem_resources&amp;gt;&amp;lt;/code&amp;gt;, &amp;lt;code&amp;gt;‑‑cpus‑per‑task=&amp;lt;cpu_amount&amp;gt;&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;‑‑time=&amp;lt;duration&amp;gt;&amp;lt;/code&amp;gt;) are omitted, the job is assigned the default amount of the resource (as defined by the chosen QOS. A notable exception concerns option &amp;lt;code&amp;gt;‑‑gres=&amp;lt;gpu_resources&amp;gt;&amp;lt;/code&amp;gt;, which is always required (see below) if the job uses a QOS with access to GPUs.&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:; --job-name=&amp;lt;jobname&amp;gt;&lt;br /&gt;
:: Specifies a name for the job. The specified name will appear along with the JOBID number when querying running jobs on the system with &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;. The default job name (i.e., the one assigned to the job when &amp;lt;code&amp;gt;--job-name&amp;lt;/code&amp;gt; is not used) is the executable program&amp;#039;s name.&lt;br /&gt;
&lt;br /&gt;
:;‑‑gres=&amp;lt;gpu_resources&amp;gt;&lt;br /&gt;
:: specifies what GPUs to assign to the job. &amp;lt;code&amp;gt;gpu_resources&amp;lt;/code&amp;gt; is a comma-delimited list where each element has the form &amp;lt;code&amp;gt;gpu:&amp;lt;Type&amp;gt;:&amp;lt;amount&amp;gt;&amp;lt;/code&amp;gt;, where &amp;lt;code&amp;gt;&amp;lt;Type&amp;gt;&amp;lt;/code&amp;gt; is one of the types of GPU available on Mufasa (see [[SLURM#gres syntax|&amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax]]) and &amp;lt;code&amp;gt;&amp;lt;amount&amp;gt;&amp;lt;/code&amp;gt; is an integer between 1 and the number of GPUs of such type available to the partition. For instance, &amp;lt;code&amp;gt;&amp;lt;gpu_resources&amp;gt;&amp;lt;/code&amp;gt; may be &amp;lt;code&amp;gt;gpu:3g.20gb:1&amp;lt;/code&amp;gt;, corresponding to asking for one &amp;quot;small&amp;quot; GPU.&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;Important! The &amp;lt;code&amp;gt;‑‑gres&amp;lt;/code&amp;gt; parameter is &amp;#039;&amp;#039;&amp;#039;mandatory&amp;#039;&amp;#039;&amp;#039; if the job is run with a QOS that allows access to the system&amp;#039;s GPUs. Differently from other resources (where unspecified requests lead to the assignment of a default amount), GPUs must always be explicitly requested.&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:;‑‑mem=&amp;lt;mem_resources&amp;gt;&lt;br /&gt;
:: specifies the amount of RAM to assign to the job; for instance, &amp;lt;code&amp;gt;&amp;lt;mem_resources&amp;gt;&amp;lt;/code&amp;gt; may be &amp;lt;code&amp;gt;32G&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
:;‑‑cpus-per-task=&amp;lt;cpu_amount&amp;gt;&lt;br /&gt;
:: specifies how many CPUs to assign to the job; for instance, &amp;lt;code&amp;gt;&amp;lt;cpu_amount&amp;gt;&amp;lt;/code&amp;gt; may be &amp;lt;code&amp;gt;2&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
:;&amp;lt;nowiki&amp;gt;‑‑time=&amp;lt;duration&amp;gt;&amp;lt;/nowiki&amp;gt;&lt;br /&gt;
:: specifies the maximum time allowed to the job to complete, in the format &amp;lt;code&amp;gt;days-hours:minutes:seconds&amp;lt;/code&amp;gt;, where &amp;lt;code&amp;gt;days&amp;lt;/code&amp;gt; is optional; for instance, &amp;lt;code&amp;gt;&amp;lt;d-hh:mm:ss&amp;gt;&amp;lt;/code&amp;gt; may be &amp;lt;code&amp;gt;72:00:00&amp;lt;/code&amp;gt;. When the time expires, the job (if still running) gets killed by SLURM.&lt;br /&gt;
&lt;br /&gt;
:;‑‑pty&lt;br /&gt;
:: specifies that the job will be interactive (this is necessary when &amp;lt;code&amp;gt;&amp;lt;command_to_run_within_container&amp;gt;&amp;lt;/code&amp;gt; is &amp;lt;code&amp;gt;/bin/bash&amp;lt;/code&amp;gt;: see [[#Interactive jobs|Interactive jobs]])&lt;br /&gt;
&lt;br /&gt;
Note that GPU resources (if needed) must always be requested explicitly. For instance, in order to execute program &amp;lt;code&amp;gt;./my_program&amp;lt;/code&amp;gt; which needs one GPU of type &amp;lt;code&amp;gt;3g.20gb&amp;lt;/code&amp;gt; with QOS &amp;lt;code&amp;gt;gpulight&amp;lt;/code&amp;gt; we can use the SLURM command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
srun --qos=gpulight --gres=gpu:3g.20gb:1 ./my_program&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
== Interactive jobs ==&lt;br /&gt;
&lt;br /&gt;
An &amp;#039;&amp;#039;&amp;#039;interactive job&amp;#039;&amp;#039;&amp;#039; is a process that use the console to interact with their user during job execution. Such a process is manually run by the user from a &amp;#039;&amp;#039;bash shell&amp;#039;&amp;#039; (i.e. a terminal session) provided by SLURM. &lt;br /&gt;
&lt;br /&gt;
In order to ask SLURM to schedule the execution of a shell where the user can subsequently run the interactive job, it is necessary to use option &amp;lt;code&amp;gt;--pty&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
For instance, to ask SLURM to run a shell with QOS &amp;lt;code&amp;gt;nogpu&amp;lt;/code&amp;gt;, the user should use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
srun --qos=nogpu --pty /bin/bash&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
By not specifying any other options, the user is telling SLURM that they want the shell spawned by SLURM to be provided with the default amount of resources associated to QOS &amp;lt;code&amp;gt;nogpu&amp;lt;/code&amp;gt;. More generally, any combination of the other [[#Options of srun and sbatch|options of srun]] can be used together with &amp;lt;code&amp;gt;--pty&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
As every other job request to SLURM, the request to run a shell must be done from the [[System#Login server|login server]]. As soon as possible (i.e., as soon as the necessary resources are available) SLURM will open (in the same terminal that the user used to launch the &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command) a bash shell, where the user will be able to run their interactive programs. &lt;br /&gt;
&lt;br /&gt;
To the user, this corresponds to the fact that the shell they were using to interact with the login server changes into a shell opened &amp;#039;&amp;#039;directly on Mufasa&amp;#039;&amp;#039;. This corresponds to the command prompt changing from&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
&amp;lt;username&amp;gt;@mufasa2-login:~$&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
to&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
&amp;lt;username&amp;gt;@mufasa2:~$&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Another way to know if the current shell is the “base” shell or one run via SLURM is to execute command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
echo $SLURM_JOB_ID&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
If no number gets printed, this means that the shell is the “base” one. If a number is printed, it is the SLURM job ID of the /bin/bash process.&lt;br /&gt;
&lt;br /&gt;
When the user does not need the SLURM-spawned shell anymore, they should close it with command (the same used for any other Linux shell)&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
exit&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
to make the resources reserved for the interactive shell free again.&lt;br /&gt;
&lt;br /&gt;
== Non-interactive jobs ==&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; commands are very complex, and it&amp;#039;s easy to forget some option or make mistakes while using them. For non-interactive jobs, there is a solution to this problem.&lt;br /&gt;
&lt;br /&gt;
When the user job is non-interactive, in fact, the &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command can be substituted with a much simpler &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;sbatch&amp;lt;/code&amp;gt; command&amp;#039;&amp;#039;&amp;#039;. As [[#Running jobs with SLURM|already explained]], &amp;lt;code&amp;gt;sbatch&amp;lt;/code&amp;gt; can make use of an &amp;#039;&amp;#039;&amp;#039;execution script&amp;#039;&amp;#039;&amp;#039; to specify all the parts of the command to be run via SLURM. So the command becomes&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sbatch &amp;lt;execution_script&amp;gt;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
An execution script is a special type of Linux script that includes &amp;#039;&amp;#039;&amp;#039;SBATCH directives&amp;#039;&amp;#039;&amp;#039;. SBATCH directives are used to specify the values of the parameters that are otherwise set in the [options] part of an &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command.&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;Note on Linux shell scripts&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
|-&lt;br /&gt;
|&amp;#039;&amp;#039;A shell script is a text file that will be run by the bash shell. In order to be acceptable as a bash script, a text file must:&lt;br /&gt;
&lt;br /&gt;
* &amp;#039;&amp;#039;have the “executable” flag set&amp;#039;&amp;#039; (see [[System#Changing file/directory ownership and permissions|here]] for details)&lt;br /&gt;
* &amp;#039;&amp;#039;have&amp;#039;&amp;#039; &amp;lt;code&amp;gt;#!/bin/bash&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;as its very first line&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;Usually, a Linux shell script is given a name ending in &amp;#039;&amp;#039;.sh,&amp;#039;&amp;#039; such as &amp;#039;&amp;#039;my_execution_script.sh&amp;#039;&amp;#039;, but this is not mandatory.&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;Within any shell script, lines preceded by &amp;lt;code&amp;gt;#&amp;lt;/code&amp;gt; are comments (with the notable exception of the initial&amp;#039;&amp;#039; &amp;lt;code&amp;gt;#!/bin/bash&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;line)&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;Use of blank lines as spacers is allowed.&amp;#039;&amp;#039;&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
An execution script is a Linux shell script composed of two parts:&lt;br /&gt;
&lt;br /&gt;
# a &amp;#039;&amp;#039;&amp;#039;preamble&amp;#039;&amp;#039;&amp;#039;,  composed of directives using which the user specifies the values to be given to parameters, each preceded by the keyword &amp;lt;code&amp;gt;SBATCH&amp;lt;/code&amp;gt;&lt;br /&gt;
# [optionally] one or more &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; commands&amp;#039;&amp;#039;&amp;#039; that launch jobs with SLURM using the parameter values specified in the preamble&lt;br /&gt;
&lt;br /&gt;
Below is an &amp;#039;&amp;#039;&amp;#039;execution script template&amp;#039;&amp;#039;&amp;#039; to be copied and pasted into your own execution script text file. &lt;br /&gt;
&lt;br /&gt;
The template includes all the options [[#Using SLURM to run a container|already described above]], plus a few additional useful ones (for instance, those that enable SLURM to send email messages to the user in correspondence to events in the lifecycle of their job). Information about all the possible options can be found in [SLURM&amp;#039;s own documentation].&lt;br /&gt;
&lt;br /&gt;
In the template below, &amp;#039;&amp;#039;&amp;#039;#SBATCH directives&amp;#039;&amp;#039;&amp;#039; are requests made to SLURM. Notice that, though #SBATCH directives have a leading &amp;quot;#&amp;quot;, that does &amp;#039;&amp;#039;not&amp;#039;&amp;#039; mean that they are comments: exactly as the &amp;lt;code&amp;gt;#!/bin/bash&amp;lt;/code&amp;gt; at the beginning of a shell script, while starting with &amp;quot;#&amp;quot;, is not a comment as well.&lt;br /&gt;
&lt;br /&gt;
Other lines in the script that begin with &amp;lt;code&amp;gt;#&amp;lt;/code&amp;gt; not followed by SBATCH are comments.&lt;br /&gt;
&lt;br /&gt;
For what concerns directive that ask for a given amount of a resource (including time), if they are missing from the execution script (or commented out) the job will be assigned the default amount of the resource.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;blockquote&amp;gt;&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;nowiki&amp;gt;#&amp;lt;/nowiki&amp;gt;!/bin/bash&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
&amp;lt;nowiki&amp;gt;#&amp;lt;/nowiki&amp;gt;----------------start of preamble----------------&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;nowiki&amp;gt;#&amp;lt;/nowiki&amp;gt;SBATCH ‑-nodes=1&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;nowiki&amp;gt;#&amp;lt;/nowiki&amp;gt;SBATCH ‑‑ntasks=1&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;nowiki&amp;gt;#&amp;lt;/nowiki&amp;gt;SBATCH ‑-partition=jobs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;nowiki&amp;gt;#&amp;lt;/nowiki&amp;gt;SBATCH ‑-qos=&amp;lt;qos_name&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;nowiki&amp;gt;#&amp;lt;/nowiki&amp;gt;SBATCH ‑‑gres=&amp;lt;gpu_resources&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;nowiki&amp;gt;#&amp;lt;/nowiki&amp;gt;SBATCH ‑‑mem=&amp;lt;mem_resources&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;nowiki&amp;gt;#&amp;lt;/nowiki&amp;gt;SBATCH ‑‑cpus-per-task=&amp;lt;cpu_amount&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;nowiki&amp;gt;#&amp;lt;/nowiki&amp;gt;SBATCH ‑‑time=&amp;lt;d-hh:mm:ss&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;nowiki&amp;gt;#&amp;lt;/nowiki&amp;gt;SBATCH ‑‑output=./&amp;lt;filename&amp;gt;-%j.out&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;lt;nowiki&amp;gt;#&amp;lt;/nowiki&amp;gt; the text file where the output of the job gets written (i.e., standard output gets redirected onto the file). &amp;quot;%j&amp;quot; is the current time.&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;nowiki&amp;gt;#&amp;lt;/nowiki&amp;gt;SBATCH ‑‑error=./&amp;lt;filename&amp;gt;-error-%j.out&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;lt;nowiki&amp;gt;#&amp;lt;/nowiki&amp;gt; the text file where any error messages generated by the job gets written (i.e., standard error gets redirected onto the file). &amp;quot;%j&amp;quot; is the current time.&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;nowiki&amp;gt;#&amp;lt;/nowiki&amp;gt;SBATCH --job-name=&amp;lt;name&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
&amp;lt;nowiki&amp;gt;#&amp;lt;/nowiki&amp;gt;----------------end of preamble----------------&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;command_to_run&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;command_to_run&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;...&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&amp;lt;/blockquote&amp;gt;&lt;br /&gt;
&lt;br /&gt;
= Executing jobs on Mufasa =&lt;br /&gt;
&lt;br /&gt;
== Key concept: use containers! ==&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;[[System#Containers|All computation on Mufasa must occur within containers]]&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
A container is a “sandbox” containing the environment where the user&amp;#039;s application operates. Parts of Mufasa&amp;#039;s filesystem can be made visible (and writable, if the user has writing permission on them: e.g., the user&amp;#039;s &amp;lt;code&amp;gt;/home&amp;lt;/code&amp;gt; directory) to the environment of the container. This allows the containerized user application to read from, and write to, Mufasa&amp;#039;s filesystem: for instance, to read data and write results.&lt;br /&gt;
&lt;br /&gt;
The system used by Mufasa to create and execute containers is &amp;#039;&amp;#039;&amp;#039;[[System#Singularity|Singularity]]&amp;#039;&amp;#039;&amp;#039;. This wiki includes [[Singularity|directions]] on preparing containers with Singularity.&lt;br /&gt;
&lt;br /&gt;
The container where a user job runs must contain all the libraries needed by the job. In fact (for maintainability and safety reasons) &amp;#039;&amp;#039;&amp;#039;no software and no libraries are installed on Mufasa 2.0&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
== Interactive and non-interactive user jobs ==&lt;br /&gt;
&lt;br /&gt;
This section explains how to execute a user job contained in a container. It considers two types of user jobs, i.e.:&lt;br /&gt;
;: Interactive user jobs&lt;br /&gt;
::: as [[#Interactive jobs|already explained]], these are jobs that require interaction with the user while they are running, via a bash shell running within the container. The shell is used to receive commands from the user and/or print output messages. For interactive user jobs, the job is usually launched manually by the user (with a command issued via the shell) after the container is in execution.&lt;br /&gt;
&lt;br /&gt;
;: Non-interactive user jobs&lt;br /&gt;
::: are the most common variety. The user prepares the container in such a way that, when in execution, the container autonomously puts the user&amp;#039;s jobs into execution. The user does not have any communication with the container while it is in execution. Executing the container and running the required programs within the container&amp;#039;s environment is done via [[#Interactive jobs|execution scripts]].&lt;br /&gt;
&lt;br /&gt;
== Using SLURM to run an interactive job on Mufasa ==&lt;br /&gt;
&lt;br /&gt;
The first step to run an interactive user job on Mufasa is to run the [[System#Containers|container]] where the job will take place. Each user is in charge of preparing the container(s) where the user&amp;#039;s jobs will be executed.&lt;br /&gt;
&lt;br /&gt;
In order to run a container via SLURM by hand, i.e. via an interactive shell, a user must first open the shell with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
srun [general_SLURM_options] ‑‑pty /bin/bash&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where [general_SLURM_options] are those [[#Options of srun and sbatch|already described above]].&lt;br /&gt;
&lt;br /&gt;
Then the user must run the container: this is done as follows.&lt;br /&gt;
&lt;br /&gt;
First, it is necessary to load the Singularity software module with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
module load amd/singularity&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
(if needed, the list of software modules available in the system can be obtained with command &amp;lt;code&amp;gt;module av&amp;lt;/code&amp;gt;).&lt;br /&gt;
&lt;br /&gt;
Then, the user must use Singularity to run the container with command (see the [[Singularity|section about Singularity]] for further details)&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
singularity run &amp;lt;repository&amp;gt;://&amp;lt;name_of_container&amp;gt;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which pulls the container from the specified repository and executes it. Possible values for &amp;lt;code&amp;gt;&amp;lt;repository&amp;gt;&amp;lt;/code&amp;gt; are:&lt;br /&gt;
&lt;br /&gt;
:: &amp;lt;code&amp;gt;docker&amp;lt;/code&amp;gt; (Docker Hub)&lt;br /&gt;
:: &amp;lt;code&amp;gt;library&amp;lt;/code&amp;gt; (Singularityhub)&lt;br /&gt;
:: &amp;lt;code&amp;gt;path/to/container&amp;lt;/code&amp;gt; if the container is local, i.e. located in the filesystem of Mufasa&lt;br /&gt;
&lt;br /&gt;
As soon as the container is in execution, the terminal window used, so far, to interact with Mufasa becomes a shell &amp;#039;&amp;#039;in the container&amp;#039;&amp;#039;. This shell belongs to the software environment of the container, and the user can use it to interact with the container&amp;#039;s own software environment and filesystem. &lt;br /&gt;
&lt;br /&gt;
It is easy to understand if a shell is open to Mufasa or to the container because in a container shell the system prompt becomes &lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
singularity&amp;gt;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
=== Interaction between container filesystem and local filesystem === &lt;br /&gt;
&lt;br /&gt;
The filesystem inside the container and the local one, i.e. Mufasa&amp;#039;s, can interact. This means that the container can access the local filesystem to read and/or write files. However, the only parts of Mufasa&amp;#039;s filesystem that can be accessed by the container are those that the user running the container has access rights to.&lt;br /&gt;
&lt;br /&gt;
As a default, the user&amp;#039;s &amp;lt;code&amp;gt;/home/username&amp;lt;/code&amp;gt; directory on Mufasa is automatically mapped onto &amp;lt;code&amp;gt;/home/username&amp;lt;/code&amp;gt; into the filesystem of the container. Whatever is done to that container directory, the changes are actually applied to the local &amp;lt;code&amp;gt;/home/username&amp;lt;/code&amp;gt; directory on Mufasa. &lt;br /&gt;
&lt;br /&gt;
The mapping of the home directory does not need to be explicitly requested. However, if the user needs (in addition to the home directory) other parts of the local filesystem of Mufasa to be mapped onto the container&amp;#039;s filesystem, this is possible by using this modified version of the &amp;lt;code&amp;gt;singularity run&amp;lt;/code&amp;gt; command:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
singularity run --bind &amp;lt;/path/to/local/directory&amp;gt;:&amp;lt;path/to/container/directory&amp;gt; &amp;lt;repository&amp;gt;://&amp;lt;name_of_container&amp;gt;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
If &amp;lt;code&amp;gt;&amp;lt;path/to/container/directory&amp;gt;&amp;lt;/code&amp;gt; does not exist in the container&amp;#039;s filesystem, it gets created by Singularity.&lt;br /&gt;
&lt;br /&gt;
=== How to know if your shell is a SLURM job ===&lt;br /&gt;
The first way to know is to look at the shell prompt: for the [[System#Login_server|login server]] (i.e., if your shell is not being run by SLURM) it should look like&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
your_username@mufasa2-login:~$ &lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
while for Mufasa (i.e., if the shell is being run by SLURM) it should look like &lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
your_username@mufasa2:~$ &lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Sometimes you may need another way to check: for instance, you may have changed your shell prompt so that it does not print the machine name. In these cases, to know if the shell you are using is run by SLURM or not you can run command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
echo $SLURM_JOB_ID&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
If the command provides an output, your shell is a SLURM job and that output is the ID of the job. If the command doesn&amp;#039;t provide any output, your shell is not a SLURM job.&lt;br /&gt;
&lt;br /&gt;
== Using SLURM to run a non-interactive job on Mufasa ==&lt;br /&gt;
&lt;br /&gt;
When the user job to be executed into a container is non-interactive, the mechanism based on an &amp;#039;&amp;#039;execution script&amp;#039;&amp;#039; already described in [[#Non-interactive jobs|Non-interactive jobs]] is employed. The command to run the script which in turn will run the container where the user job takes place is therefore&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sbatch &amp;lt;execution_script&amp;gt;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The general features of a SLURM execution script and the SBATCH directives used for generic jobs have [[#Non-interactive jobs|already been described]]. Here we focus, therefore, on the SBATCH directives specifically used when SLURM is used to run a non-interactive job within a container.&lt;br /&gt;
&lt;br /&gt;
Below is an &amp;#039;&amp;#039;&amp;#039;execution script template&amp;#039;&amp;#039;&amp;#039; to be copied and pasted into your own execution script text file. &lt;br /&gt;
&lt;br /&gt;
&amp;lt;blockquote&amp;gt;&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;nowiki&amp;gt;#&amp;lt;/nowiki&amp;gt;!/bin/bash&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
&amp;lt;nowiki&amp;gt;#&amp;lt;/nowiki&amp;gt;----------------start of preamble----------------&lt;br /&gt;
&lt;br /&gt;
[[#Non-interactive jobs|#SBATCH directives already described above]]&lt;br /&gt;
&lt;br /&gt;
&amp;lt;nowiki&amp;gt;#&amp;lt;/nowiki&amp;gt;----------------end of preamble----------------&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;module load amd/singularity&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;singularity run &amp;lt;repository&amp;gt;://&amp;lt;name_of_container&amp;gt; &amp;lt;command_to_run&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
&amp;lt;/blockquote&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In the last line of the script, &amp;lt;code&amp;gt;&amp;lt;command_to_run&amp;gt;&amp;lt;/code&amp;gt; is the command (e.g., the name of an executable script), complete with path within the container&amp;#039;s filesystem, of the program to be run into the container. Please refer to the [[Singularity|section about Singularity]] for details about its commands.&lt;br /&gt;
&lt;br /&gt;
The interactions between container filesystem and local filesystem in non-interactive jobs are exactly the same [[#Interaction between container filesystem and local filesystem|already described]] for interactive jobs. In particular, the user&amp;#039;s home directory is mapped by default onto the filesystem of the container.&lt;br /&gt;
&lt;br /&gt;
If, in addition to that, the user needs another part of the filesystem of Mufasa are to be mapped onto the container&amp;#039;s filesystem, this is possible using this modified version of the &amp;lt;code&amp;gt;singularity run&amp;lt;/code&amp;gt; command at the end of the script:&lt;br /&gt;
&lt;br /&gt;
:&amp;lt;code&amp;gt;singularity run --bind &amp;lt;/path/to/local/directory&amp;gt;:&amp;lt;path/to/container/directory&amp;gt; &amp;lt;repository&amp;gt;://&amp;lt;name_of_container&amp;gt;&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
If &amp;lt;code&amp;gt;&amp;lt;path/to/container/directory&amp;gt;&amp;lt;/code&amp;gt; does not exist in the container&amp;#039;s filesystem, it gets created by Singularity.&lt;br /&gt;
&lt;br /&gt;
== Job output ==&lt;br /&gt;
&lt;br /&gt;
The whole point of running a user job is to collect its output. Usually, such output takes the form of one or more files generated within the filesystem of Mufasa by the container where the computation takes place. &lt;br /&gt;
&lt;br /&gt;
As [[#Using SLURM to run a container|explained below]], SLURM includes a mechanism to mount a part of Mufasa&amp;#039;s own filesystem onto the container&amp;#039;s filesystem: so when the job running within the container writes to this mounted part, it actually writes to Mufasa&amp;#039;s filesystem. This means that when the container ends its execution, its output files persist in Mufasa&amp;#039;s filesystem (usually in a subdirectory of the user&amp;#039;s own &amp;lt;code&amp;gt;/home&amp;lt;/code&amp;gt; directory) and can be retrieved by the user at a later time.&lt;br /&gt;
&lt;br /&gt;
The same mechanism can be used to allow user jobs running into a container to read their input data from Mufasa&amp;#039;s filesystem (usually a subdirectory of the user&amp;#039;s own &amp;lt;code&amp;gt;/home&amp;lt;/code&amp;gt; directory).&lt;br /&gt;
&lt;br /&gt;
== Cancelling completed jobs ==&lt;br /&gt;
&lt;br /&gt;
When a user process run via SLURM has completed its execution and is not needed anymore, it is important to [[#Cancelling a job with scancel|close it with scancel]]. Especially if much time remains to the end of the execution time requested by the job.&lt;br /&gt;
&lt;br /&gt;
Cancelling a SLURM job makes the resources reserved by SLURM free again for other users, and thus speeds up the execution of the jobs still queued.&lt;br /&gt;
&lt;br /&gt;
Typically, one doesn&amp;#039;t know how long a piece of code will take to complete its work. So please make sure to check from time to time if that happened, and -if there&amp;#039;s still time before the duration of your SLURM job ends- just &amp;#039;&amp;#039;scancel&amp;#039;&amp;#039; the job. Other users will be grateful :-)&lt;br /&gt;
&lt;br /&gt;
Please note that [[System#Job priority|job priority]] for your user depends (also) on the overall duration of the jobs that you ran on Mufasa. Therefore, &amp;#039;&amp;#039;&amp;#039;cancelling jobs that are not needed anymore improves your future jobs&amp;#039; priority&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
= Looking for unused GPUs =&lt;br /&gt;
&lt;br /&gt;
GPUs are usually the most limited resource on Mufasa. So, if your job requires a GPU, the best way to get it executed quickly is to use a QOS associated to a type of GPU of which there are one or more that aren&amp;#039;t currently in use. This command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sinfo -O Gres:100&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
provides a summary of all the Gres (i.e., GPU) resources possessed by Mufasa. It provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
GRES                                                                                                &lt;br /&gt;
gpu:40gb:3,gpu:4g.20gb:5,gpu:3g.20gb:5&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
To know which of the GPUs are currently in use, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sinfo -O GresUsed:100&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
GRES_USED&lt;br /&gt;
gpu:40gb:2(IDX:0-1),gpu:4g.20gb:2(IDX:5,8),gpu:3g.20gb:3(IDX:3-4,6)&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
By comparing the two lists (GRES and GRES_USED) you can easily spot unused GPUs.&lt;br /&gt;
&lt;br /&gt;
= Detaching from a running job with &amp;lt;code&amp;gt;screen&amp;lt;/code&amp;gt; =&lt;br /&gt;
&lt;br /&gt;
A consequence of the way &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; operates is that if you launch an [[#Interactive and non-interactive user jobs|interactive user job]], the shell where the command is running must remain open: if it closes, the job terminates. That shell runs in the terminal of your own PC where the [[System#Accessing Mufasa|SSH connection to Mufasa]] exists.&lt;br /&gt;
&lt;br /&gt;
If you do not plan to keep the SSH connection to Mufasa open (for instance because you have to turn off or suspend your PC), there is a way to keep your interactive job alive. Namely, you should use command &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; inside a &amp;#039;&amp;#039;screen session&amp;#039;&amp;#039; (often simply called &amp;quot;a screen&amp;quot;), then &amp;#039;&amp;#039;detach&amp;#039;&amp;#039; from the &amp;#039;&amp;#039;screen&amp;#039;&amp;#039; ([https://linuxize.com/post/how-to-use-linux-screen/ here] is one of many tutorials about &amp;lt;code&amp;gt;screen&amp;lt;/code&amp;gt; available online). &lt;br /&gt;
&lt;br /&gt;
Once you have detached from the screen session, you can close the SSH connection to Mufasa without damage. When you need to reach your (still running) job again, you can can open a new SSH connection to Mufasa and then &amp;#039;&amp;#039;reattach&amp;#039;&amp;#039; to the &amp;#039;&amp;#039;screen&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
A use case for screen is writing your program in such a way that it prints progress advancement messages as it goes on with its work. Then, you can check its advancement by periodically reconnecting to the screen where the program is running and reading the messages it printed.&lt;br /&gt;
&lt;br /&gt;
Basic usage of &amp;lt;code&amp;gt;screen&amp;lt;/code&amp;gt; is explained below.&lt;br /&gt;
&lt;br /&gt;
== Creating a screen session, running a job in it, detaching from it ==&lt;br /&gt;
&lt;br /&gt;
# Connect to the [[System#Login server|login server]] with SSH&lt;br /&gt;
# From the login server shell, run &amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;screen&amp;lt;/pre&amp;gt;&lt;br /&gt;
# In the &amp;#039;&amp;#039;screen session&amp;#039;&amp;#039; (&amp;quot;screen&amp;quot;) thus created (it has the look of an empty shell), launch your job with &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt;&lt;br /&gt;
# &amp;#039;&amp;#039;Detach&amp;#039;&amp;#039; from the screen by pressing &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;ctrl + A&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039; followed by &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;D&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;: you will come back to the original login server shell, while your process will go on running in the screen&lt;br /&gt;
# You can now close the SSH connection to the login server without damaging your running job&lt;br /&gt;
&lt;br /&gt;
== Reattaching to an active screen session ==&lt;br /&gt;
&lt;br /&gt;
# Connect to the [[System#Login server|login server]] with SSH&lt;br /&gt;
# In the login server shell, run &amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;screen -r&amp;lt;/pre&amp;gt;&lt;br /&gt;
# You are now back to the screen where you launched your job&lt;br /&gt;
&lt;br /&gt;
== Closing (i.e. destroying) a screen session ==&lt;br /&gt;
&lt;br /&gt;
When you do not need a screen session anymore:&lt;br /&gt;
&lt;br /&gt;
# reattach to the active screen session as explained [[#Reattaching to an active screen session|above]]&lt;br /&gt;
# destroy the screen by pressing &amp;#039;&amp;#039;&amp;#039;ctrl + A&amp;#039;&amp;#039;&amp;#039; followed by &amp;#039;&amp;#039;&amp;#039;\&amp;#039;&amp;#039;&amp;#039; (i.e., backslash), then confirming that you really want to proceed&lt;br /&gt;
&lt;br /&gt;
Of course, any program (including SLURM jobs) running within the screen gets terminated when the screen is destroyed.&lt;br /&gt;
&lt;br /&gt;
= Using &amp;lt;code&amp;gt;salloc&amp;lt;/code&amp;gt; to reserve resources =&lt;br /&gt;
&lt;br /&gt;
== What is &amp;lt;code&amp;gt;salloc&amp;lt;/code&amp;gt;? ==&lt;br /&gt;
&lt;br /&gt;
[https://slurm.schedmd.com/salloc.html &amp;lt;code&amp;gt;salloc&amp;lt;/code&amp;gt;] is a SLURM command that allows a user to reserve a set of resources (e.g., a 40 GB GPU) for a given time in the future.&lt;br /&gt;
&lt;br /&gt;
The typical use of &amp;lt;code&amp;gt;salloc&amp;lt;/code&amp;gt; is to &amp;quot;book&amp;quot; an interactive session where the user enjoys &amp;#039;&amp;#039;&amp;#039;complete control of a set of resources&amp;#039;&amp;#039;&amp;#039;. The resources that are part of this set are chosen by the user. Within the &amp;quot;booked&amp;quot; session, any job run by the user that relies on the reserved resources is immediately put into execution by SLURM.&lt;br /&gt;
&lt;br /&gt;
More precisely:&lt;br /&gt;
* the user, using &amp;lt;code&amp;gt;salloc&amp;lt;/code&amp;gt;, specifies what resources they need and the time when they will need them;&lt;br /&gt;
* when the delivery comes, SLURM creates an interactive shell session for the user;&lt;br /&gt;
* within such session, the user can use &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;sbatch&amp;lt;/code&amp;gt; to run programs, enjoying full (i.e. not shared with anyone else) and instantaneous access to the resources.&lt;br /&gt;
&lt;br /&gt;
Resource reservation using &amp;lt;code&amp;gt;salloc&amp;lt;/code&amp;gt; is only possible if the request is done in advance wrt the delivery time. The more the resources that the user wants to reserve are in high demand, the more anticipated the request should be to ensure that SLURM is able to fulfill it.&lt;br /&gt;
&lt;br /&gt;
When a user makes a request for resources with &amp;lt;code&amp;gt;salloc&amp;lt;/code&amp;gt;, the request (called an &amp;#039;&amp;#039;&amp;#039;allocation&amp;#039;&amp;#039;&amp;#039;) gets added to the job queue of SLURM of the requisite partition as a job in &amp;lt;code&amp;gt;pending&amp;lt;/code&amp;gt; (&amp;lt;code&amp;gt;PD&amp;lt;/code&amp;gt;) state (job states are described [[User_Jobs#Interpreting Job state as provided by squeue|here]]). Indeed, resource allocation is the first part of SLURM&amp;#039;s process of executing a user job, while the second part is running the program and letting it use the allocated resources. Using &amp;lt;code&amp;gt;salloc&amp;lt;/code&amp;gt; actually corresponds to having SLURM perform the first part of the process (resource allocation) while leaving the second part (running programs) to the user.&lt;br /&gt;
&lt;br /&gt;
Until the delivery time specified by the user comes, the allocation remains in state &amp;lt;code&amp;gt;PD&amp;lt;/code&amp;gt;, and other jobs requesting the same resources, even if submitted later, are executed. While the request waits for the delivery time, however, it accumulates a priority that increases over time. The longer the allocation stays in the &amp;lt;code&amp;gt;PD&amp;lt;/code&amp;gt; state, the stronger this accumulation of priority: so, by requesting resources with &amp;lt;code&amp;gt;salloc&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;&amp;#039;well in advance of the delivery time&amp;#039;&amp;#039;&amp;#039;, users can ensure that the resources they need will be ready for them at the requested delivery time, even if these resources are highly contended.&lt;br /&gt;
&lt;br /&gt;
== &amp;lt;code&amp;gt;salloc&amp;lt;/code&amp;gt; commands ==&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;salloc&amp;lt;/code&amp;gt; commands use a similar syntax to &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; commands. In particular, &amp;lt;code&amp;gt;salloc&amp;lt;/code&amp;gt; lets a user specify what resources they need and -importantly- a &amp;#039;&amp;#039;&amp;#039;delivery time&amp;#039;&amp;#039;&amp;#039; for the requested resources (delivery time can also be specified with &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt;, but in that case it is not very useful). &lt;br /&gt;
&lt;br /&gt;
The typical &amp;lt;code&amp;gt;salloc&amp;lt;/code&amp;gt; command has this form:&amp;#039;&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
salloc [general_SLURM_options] --begin=&amp;lt;time&amp;gt;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where &lt;br /&gt;
&lt;br /&gt;
:; [general_SLURM_options]&lt;br /&gt;
:: represents the options already described in [[#Options of srun and sbatch|Options of srun and sbatch]]&lt;br /&gt;
&lt;br /&gt;
:;&amp;lt;nowiki&amp;gt;--begin=&amp;lt;time&amp;gt;&amp;lt;/nowiki&amp;gt;&lt;br /&gt;
:: specifies the delivery time of the resources reserved with &amp;lt;code&amp;gt;salloc&amp;lt;/code&amp;gt;, according to the syntax described below. The delivery time must be a future time.&lt;br /&gt;
&lt;br /&gt;
=== Syntax of parameter &amp;lt;code&amp;gt;--begin&amp;lt;/code&amp;gt; ===&lt;br /&gt;
&lt;br /&gt;
If the allocation is for the current day, you can specify &amp;lt;nowiki&amp;gt;&amp;lt;time&amp;gt;&amp;lt;/nowiki&amp;gt; as hours and minutes in the form&lt;br /&gt;
&lt;br /&gt;
:&amp;lt;code&amp;gt;HH:MM&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
If you want to specify a time of a different day, the form for &amp;lt;time&amp;gt; is &amp;lt;code&amp;gt;YYYY-MM-DDTHH:MM&amp;lt;/code&amp;gt;, where the uppercase &amp;#039;T&amp;#039; separates date from time. &lt;br /&gt;
&lt;br /&gt;
It is also possible to specify &amp;lt;time&amp;gt; as relative to the current time, in one of the following forms:&lt;br /&gt;
: &amp;lt;code&amp;gt;now+Kminutes&amp;lt;/code&amp;gt;&lt;br /&gt;
: &amp;lt;code&amp;gt;now+Khours&amp;lt;/code&amp;gt;&lt;br /&gt;
: &amp;lt;code&amp;gt;now+Kdays&amp;lt;/code&amp;gt;&lt;br /&gt;
where K is a (positive) integer.&lt;br /&gt;
&lt;br /&gt;
Examples:&lt;br /&gt;
: &amp;lt;code&amp;gt;--begin=16:00&amp;lt;/code&amp;gt;&lt;br /&gt;
: &amp;lt;code&amp;gt;--begin=now+1hours&amp;lt;/code&amp;gt;&lt;br /&gt;
: &amp;lt;code&amp;gt;--begin=now+1days&amp;lt;/code&amp;gt;&lt;br /&gt;
: &amp;lt;code&amp;gt;--begin=2030-01-20T12:34:00&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Note that Mufasa&amp;#039;s time zone is GMT, so &amp;lt;nowiki&amp;gt;&amp;lt;time&amp;gt;&amp;lt;/nowiki&amp;gt; must be expressed in GMT as well. If you want to know Mufasa&amp;#039;s current time, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
date&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
It provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
Thu Nov 10 16:43:30 UTC 2022&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
== How to use &amp;lt;code&amp;gt;salloc&amp;lt;/code&amp;gt; ==&lt;br /&gt;
&lt;br /&gt;
In the typical scenario, the user of &amp;lt;code&amp;gt;salloc&amp;lt;/code&amp;gt; will make use of [[User_Jobs#Detaching from a running job with screen|screen]]. Command &amp;lt;code&amp;gt;screen&amp;lt;/code&amp;gt; creates a shell session (called &amp;quot;a screen&amp;quot;) that it is possible to abandon without closing it ([[#Creating_a_screen_session.2C_running_a_job_in_it.2C_detaching_from_it|detaching from the screen]]). It is then possible to reach again the screen at a later time ([[#Reattaching_to_an_active_screen_session|reattaching to the screen]]). This means that a user can create a screen, run &amp;lt;code&amp;gt;salloc&amp;lt;/code&amp;gt; within it to create an allocation for time X, detach from the screen and reattach to it just before time X to use the reserved resources from the interactive session created by &amp;lt;code&amp;gt;salloc&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
More precisely, the operations needed to do this are the following:&lt;br /&gt;
&lt;br /&gt;
# Connect to the [[System#Login server|login server]].&lt;br /&gt;
# From the login server shell, run &amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;screen&amp;lt;/pre&amp;gt;&lt;br /&gt;
# In the &amp;#039;&amp;#039;screen session&amp;#039;&amp;#039; (&amp;quot;screen&amp;quot;) thus created run the [[#salloc commands|&amp;lt;code&amp;gt;salloc&amp;lt;/code&amp;gt; command]], specifying via its options the resources you need and the time at which you want them delivered.&lt;br /&gt;
# SLURM will respond with a message similar to &amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;salloc: Pending job allocation XXXX&amp;lt;/pre&amp;gt;&lt;br /&gt;
# &amp;#039;&amp;#039;Detach&amp;#039;&amp;#039; from the screen by pressing &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;ctrl + A&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039; followed by &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;D&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;: you will come back to the original login server shell.&lt;br /&gt;
# You can now close the SSH connection to the login server without damaging your resource allocation request.&lt;br /&gt;
# At the delivery time you specified in the [[#salloc commands|&amp;lt;code&amp;gt;salloc&amp;lt;/code&amp;gt; command]], connect to the login server with SSH.&lt;br /&gt;
# Once you are in the login server shell, reattach to the screen with command &amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;screen -r&amp;lt;/pre&amp;gt;&lt;br /&gt;
# You are now back to the screen where you used &amp;lt;code&amp;gt;salloc&amp;lt;/code&amp;gt;; as soon as SLURM provides to you with the resources you reserved, message &amp;quot;&amp;#039;&amp;#039;salloc: Pending job allocation XXXX&amp;#039;&amp;#039;&amp;quot; changes to the shell prompt.&lt;br /&gt;
# You are now in the interactive shell session you booked with &amp;lt;code&amp;gt;salloc&amp;lt;/code&amp;gt;. From here, you can run any programs you want, including &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;sbatch&amp;lt;/code&amp;gt;. For the whole duration of the allocation, your programs have unrestricted use of all the resources you reserved with &amp;lt;code&amp;gt;salloc&amp;lt;/code&amp;gt;.&amp;lt;br&amp;gt;&amp;#039;&amp;#039;&amp;#039;Important!&amp;#039;&amp;#039;&amp;#039; Any job run within the shell session is subject to the time limit (i.e., maximum duration) imposed by the partition it is running on! Therefore, if the job reaches the time limit, it gets &amp;#039;&amp;#039;&amp;#039;forcibly terminated&amp;#039;&amp;#039;&amp;#039; by SLURM. Termination depends exclusively from the time limit: so it occurs even if the end time for the allocation has not been reached yet. (Of course, the job also gets terminated if the allocation ends.)&lt;br /&gt;
# Once the interactive shell session is not needed anymore, cancel it by exiting from the session with &amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;exit&amp;lt;/pre&amp;gt; (Note that if you get to the end of the time period you specified in your request without closing the shell session, SLURM does it for you, killing any programs still running.)&lt;br /&gt;
# You are now back to your screen. Destroy it by pressing &amp;#039;&amp;#039;&amp;#039;ctrl + A&amp;#039;&amp;#039;&amp;#039; followed by &amp;#039;&amp;#039;&amp;#039;\&amp;#039;&amp;#039;&amp;#039; (i.e., backslash) to get back to the login server shell.&lt;br /&gt;
&lt;br /&gt;
== Cancelling a resource request made with &amp;lt;code&amp;gt;salloc&amp;lt;/code&amp;gt; ==&lt;br /&gt;
&lt;br /&gt;
To cancel a request for resources made as explained in [[#How to use salloc|How to use &amp;lt;code&amp;gt;salloc&amp;lt;/code&amp;gt;]], follow these steps:&lt;br /&gt;
&lt;br /&gt;
# Connect to the the [[System#Login server|login server]] with SSH.&lt;br /&gt;
# Once you are in the login server shell, reattach to the screen where you used command &amp;lt;code&amp;gt;salloc&amp;lt;/code&amp;gt; with command &amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;screen -r&amp;lt;/pre&amp;gt;&lt;br /&gt;
# You should see the message &amp;quot;&amp;#039;&amp;#039;salloc: Pending job allocation XXXX&amp;#039;&amp;#039;&amp;quot; (if the allocation is still pending) or &amp;quot;&amp;quot;&amp;#039;&amp;#039;salloc: job XXXX queued and waiting for resources&amp;#039;&amp;#039;&amp;quot; (if the allocation is done and waiting for its start time). Now just press &amp;#039;&amp;#039;&amp;#039;Ctrl + C&amp;#039;&amp;#039;&amp;#039;. This communicates to SLURM your intention to cancel your request for resources.&lt;br /&gt;
# SLURM will communicate the cancellation with message &amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;salloc: Job allocation XXXX has been revoked.&amp;lt;/pre&amp;gt;&lt;br /&gt;
# Destroy the screen by pressing &amp;#039;&amp;#039;&amp;#039;ctrl + A&amp;#039;&amp;#039;&amp;#039; followed by &amp;#039;&amp;#039;&amp;#039;\&amp;#039;&amp;#039;&amp;#039; (i.e., backslash) to get back to the login server shell.&lt;br /&gt;
&lt;br /&gt;
= Monitoring and managing jobs =&lt;br /&gt;
&lt;br /&gt;
SLURM provides Job Users with tools to inspect and manage jobs. While a [[Roles|Job User]] is able to see all users&amp;#039; jobs, they are only allowed to interact with their own.&lt;br /&gt;
&lt;br /&gt;
The main commands used to interact with jobs are &amp;#039;&amp;#039;&amp;#039;[https://slurm.schedmd.com/squeue.html &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;]&amp;#039;&amp;#039;&amp;#039; to inspect the scheduling queues and &amp;#039;&amp;#039;&amp;#039;[https://slurm.schedmd.com/scancel.html &amp;lt;code&amp;gt;scancel&amp;lt;/code&amp;gt;]&amp;#039;&amp;#039;&amp;#039; to terminate queued or running jobs.&lt;br /&gt;
&lt;br /&gt;
== Inspecting jobs with &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt; ==&lt;br /&gt;
&lt;br /&gt;
Running command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
squeue&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)&lt;br /&gt;
  520       fat     bash acasella  R 2-04:10:25      1 gn01&lt;br /&gt;
  523       fat     bash amarzull  R    1:30:35      1 gn01&lt;br /&gt;
  522       gpu     bash    clena  R   20:51:16      1 gn01&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This output comprises the following information:&lt;br /&gt;
&lt;br /&gt;
:; JOBID&lt;br /&gt;
:: Numerical identifier of the job assigned by SLURM&lt;br /&gt;
:: This identifier is used to intervene on the job, for instance with &amp;lt;code&amp;gt;scancel&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
:; PARTITION&lt;br /&gt;
:: the partition that the job is run on&lt;br /&gt;
&lt;br /&gt;
:; NAME&lt;br /&gt;
:: the name assigned to the job; can be personalised using the &amp;lt;code&amp;gt;--job-name&amp;lt;/code&amp;gt; option&lt;br /&gt;
&lt;br /&gt;
:; USER&lt;br /&gt;
:: username of the user who launched the job&lt;br /&gt;
	&lt;br /&gt;
:; ST&lt;br /&gt;
:: job state (see [[SLURM#Job state|Job state]] for further information)&lt;br /&gt;
&lt;br /&gt;
:; TIME&lt;br /&gt;
:: time that has passed since the beginning of job execution&lt;br /&gt;
&lt;br /&gt;
:; NODES&lt;br /&gt;
:: number of nodes where the job is being executed (for Mufasa, this is always 1 as it is a single machine)&lt;br /&gt;
&lt;br /&gt;
:; NODELIST (REASON)&lt;br /&gt;
:: name of the nodes where the job is being executed: for Mufasa it is always &amp;lt;code&amp;gt;gn01&amp;lt;/code&amp;gt;, which is the name of the node corresponding to Mufasa.&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
To limit the output of &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt; to the jobs owned by user &amp;lt;code&amp;gt;&amp;lt;username&amp;gt;&amp;lt;/code&amp;gt;, it can be used like this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
squeue -u &amp;lt;username&amp;gt;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
=== Interpreting Job state as provided by &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt; ===&lt;br /&gt;
&lt;br /&gt;
Jobs typically pass through several states in the course of their execution. Job state is shown in column &amp;quot;ST&amp;quot; of the output of &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt; as an abbreviated code (e.g., &amp;quot;R&amp;quot; for RUNNING).&lt;br /&gt;
&lt;br /&gt;
The most relevant codes and states are the following:&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;PD&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; PENDING&lt;br /&gt;
:: Job is awaiting resource allocation. &lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;R&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; RUNNING&lt;br /&gt;
:: Job currently has an allocation.&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;S&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; SUSPENDED&lt;br /&gt;
:: Job has an allocation, but execution has been suspended and CPUs have been released for other jobs. &lt;br /&gt;
 &lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;CG&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; COMPLETING&lt;br /&gt;
:: Job is in the process of completing. Some processes on some nodes may still be active. &lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;CD&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; COMPLETED&lt;br /&gt;
:: Job has terminated all processes on all nodes with an exit code of zero. &lt;br /&gt;
&lt;br /&gt;
Beyond these, there are other (less frequent) job states. [https://slurm.schedmd.com/squeue.html The SLURM doc page for &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;] provides a complete list of them.&lt;br /&gt;
&lt;br /&gt;
== Knowing when jobs are expected to end or start ==&lt;br /&gt;
&lt;br /&gt;
If you are interested in understanding when jobs are expected to start or end, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
squeue -o &amp;quot;%5i %8u %10P %.2t |%19S |%.11L|&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output is similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
JOBID USER     PARTITION  ST |START_TIME          |  TIME_LEFT|&lt;br /&gt;
5307  thuynh   fat        PD |2022-11-11T17:55:54 | 3-00:00:00|&lt;br /&gt;
5308  thuynh   fat        PD |2022-11-11T17:55:54 | 3-00:00:00|&lt;br /&gt;
5296  cziyang  fat         R |2022-11-08T16:58:03 | 1-00:48:14|&lt;br /&gt;
5306  thuynh   fat         R |2022-11-10T08:13:30 | 2-16:03:41|&lt;br /&gt;
5297  gnannini fat         R |2022-11-08T17:55:54 | 1-01:46:05|&lt;br /&gt;
5336  ssaitta  gpu         R |2022-11-10T08:13:00 |    6:03:11|&lt;br /&gt;
5358  dmilesi  gpulong     R |2022-11-10T15:11:32 | 2-23:01:43|&lt;br /&gt;
5338  cziyang  gpulong     R |2022-11-10T09:45:01 | 1-17:35:12|&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
;:For running jobs (state &amp;lt;code&amp;gt;R&amp;lt;/code&amp;gt;):&lt;br /&gt;
::column &amp;quot;START_TIME&amp;quot; tells you when the job started its execution&lt;br /&gt;
::column &amp;quot;TIME_LEFT&amp;quot; tells you how much remains of the running time requested by the job&lt;br /&gt;
&lt;br /&gt;
;:For pending jobs (state &amp;lt;code&amp;gt;PD&amp;lt;/code&amp;gt;):&lt;br /&gt;
::column &amp;quot;START_TIME&amp;quot; tells you when the job is expected to start its execution&lt;br /&gt;
::column &amp;quot;TIME_LEFT&amp;quot; tells you how much running time has been requested by the job&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;Important!&amp;#039;&amp;#039;&amp;#039; Start and end times are forecasts based on the features of current jobs in the queues, and may change if running jobs end prematurely and/or if new jobs with higher priority are added to the queues. So these times should never be considered as certain.&lt;br /&gt;
&lt;br /&gt;
If you simply want to know when pending jobs (state &amp;lt;code&amp;gt;PD&amp;lt;/code&amp;gt;) are expected to begin execution, use&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
squeue --start&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which lists pending jobs in order of increasing START_TIME (the job on top is the one which will be run first). For each pending job the command provides an output similar to the example below:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
JOBID PARTITION     NAME     USER ST          START_TIME  NODES SCHEDNODES           NODELIST(REASON)&lt;br /&gt;
 5090       fat training   thuynh PD 2022-10-27T09:28:01      1 (null)               (Resources)&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
== Getting detailed information about a job ==&lt;br /&gt;
&lt;br /&gt;
If needed, complete information about a job (either pending or running) can be obtained using command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scontrol show job &amp;lt;JOBID&amp;gt;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where &amp;lt;code&amp;gt;&amp;lt;JOBID&amp;gt;&amp;lt;/code&amp;gt; is the number from the first column of the output of &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;. The output of this command is similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
JobId=65 JobName=test_script.sh&lt;br /&gt;
   UserId=gfontana(10003) GroupId=gfontana(10004) MCS_label=N/A&lt;br /&gt;
   Priority=14208 Nice=0 Account=admin QOS=nogpu&lt;br /&gt;
   JobState=RUNNING Reason=None Dependency=(null)&lt;br /&gt;
   Requeue=0 Restarts=0 BatchFlag=0 Reboot=0 ExitCode=0:0&lt;br /&gt;
   RunTime=00:00:55 TimeLimit=01:00:00 TimeMin=N/A&lt;br /&gt;
   SubmitTime=2025-11-06T10:31:10 EligibleTime=2025-11-06T10:31:10&lt;br /&gt;
   AccrueTime=2025-11-06T10:31:10&lt;br /&gt;
   StartTime=2025-11-06T10:31:10 EndTime=2025-11-06T11:31:10 Deadline=N/A&lt;br /&gt;
   SuspendTime=None SecsPreSuspend=0 LastSchedEval=2025-11-06T10:31:10 Scheduler=Main&lt;br /&gt;
   Partition=jobs AllocNode:Sid=mufasa2-login:42020&lt;br /&gt;
   ReqNodeList=(null) ExcNodeList=(null)&lt;br /&gt;
   NodeList=gn01&lt;br /&gt;
   BatchHost=gn01&lt;br /&gt;
   NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:*&lt;br /&gt;
   ReqTRES=cpu=1,mem=4G,node=1,billing=1&lt;br /&gt;
   AllocTRES=cpu=1,mem=4G,node=1,billing=1&lt;br /&gt;
   Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*&lt;br /&gt;
   MinCPUsNode=1 MinMemoryNode=4G MinTmpDiskNode=0&lt;br /&gt;
   Features=(null) DelayBoot=00:00:00&lt;br /&gt;
   OverSubscribe=OK Contiguous=0 Licenses=(null) LicensesAlloc=(null) Network=(null)&lt;br /&gt;
   Command=./test_script.sh&lt;br /&gt;
   WorkDir=/home/gfontana&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In particular, the line beginning with &amp;#039;&amp;#039;&amp;quot;StartTime=&amp;quot;&amp;#039;&amp;#039; provides expected times for the start and end of job execution. As explained in [[User_Jobs#Knowing_when_jobs_are_expected_to_end_or_start|Knowing when jobs are expected to end or start]], start time is only a prediction and subject to change.&lt;br /&gt;
&lt;br /&gt;
== Cancelling a job with &amp;lt;code&amp;gt;scancel&amp;lt;/code&amp;gt; ==&lt;br /&gt;
&lt;br /&gt;
It is possible to cancel a job using command &amp;lt;code&amp;gt;scancel&amp;lt;/code&amp;gt;, either while it is waiting for execution or when it is in execution (in this case you can choose what system signal to send the process in order to terminate it). &lt;br /&gt;
&lt;br /&gt;
Please note that [[System#Job priority|job priority]] for your user depends (also) on the overall duration of the jobs that you ran on Mufasa. Therefore, &amp;#039;&amp;#039;&amp;#039;cancelling jobs that are not needed anymore improves your future jobs&amp;#039; priority&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
The following are some examples of use of &amp;lt;code&amp;gt;scancel&amp;lt;/code&amp;gt; adapted from [https://slurm.schedmd.com/scancel.html SLURM&amp;#039;s documentation].&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scancel &amp;lt;JOBID&amp;gt;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
removes queued job &amp;lt;code&amp;gt;&amp;lt;JOBID&amp;gt;&amp;lt;/code&amp;gt; from the execution queue.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scancel --signal=TERM &amp;lt;JOBID&amp;gt;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
terminates execution of job &amp;lt;code&amp;gt;&amp;lt;JOBID&amp;gt;&amp;lt;/code&amp;gt; with signal SIGTERM (request to stop).&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scancel --signal=KILL &amp;lt;JOBID&amp;gt;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
terminates execution of job &amp;lt;code&amp;gt;&amp;lt;JOBID&amp;gt;&amp;lt;/code&amp;gt; with signal SIGKILL (force stop).&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scancel --state=PENDING --user=&amp;lt;username&amp;gt; --partition=&amp;lt;partition_name&amp;gt;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
cancels all pending jobs belonging to user &amp;lt;code&amp;gt;&amp;lt;username&amp;gt;&amp;lt;/code&amp;gt; in partition &amp;lt;code&amp;gt;&amp;lt;partition_name&amp;gt;&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
== Knowing what jobs you ran today ==&lt;br /&gt;
&lt;br /&gt;
Command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacct -X --format=User%-10,Start,End,SubmitLine%-100&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
provides a list of all &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;sbatch&amp;lt;/code&amp;gt; commands executed by your user after yesterday&amp;#039;s midnight. For each job, the command provides the start and end time and the command you used to run it (if needed, you can increase the value after &amp;lt;code&amp;gt;SubmitLine%-&amp;lt;/code&amp;gt; to show longer commands in full).&lt;/div&gt;</summary>
		<author><name>GiulioFontana</name></author>
	</entry>
	<entry>
		<id>https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2592</id>
		<title>SLURM</title>
		<link rel="alternate" type="text/html" href="https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2592"/>
		<updated>2026-11-04T09:13:06Z</updated>

		<summary type="html">&lt;p&gt;GiulioFontana: /* Elements determining job priority */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page presents the features of SLURM that are most relevant to Mufasa&amp;#039;s [[Roles|Job Users]]. Job Users can submit jobs for execution, cancel their own jobs, and see other users&amp;#039; jobs (but not intervene on them).&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa &amp;#039;&amp;#039;&amp;#039;must use SLURM&amp;#039;&amp;#039;&amp;#039; to run resource-heavy processes, i.e. computing jobs that require one or more of the following:&lt;br /&gt;
* GPUs&lt;br /&gt;
* multiple CPUs&lt;br /&gt;
* powerful CPUs&lt;br /&gt;
* a significant amount of RAM&lt;br /&gt;
&lt;br /&gt;
In fact, only processes run via SLURM have access to all the resources of Mufasa. Processes run outside SLURM are executed by the [[System#Login server|login server]] virtual machine, which has minimal resources and no GPUs. Using SLURM is therefore the only way to execute resource-heavy jobs on Mufasa (this is a key difference between Mufasa 1.0 and Mufasa 2.0).&lt;br /&gt;
&lt;br /&gt;
= SLURM in a nutshell =&lt;br /&gt;
&lt;br /&gt;
Computation jobs on Mufasa needs to be launched via [[System#The SLURM job scheduling system|SLURM]]. SLURM provides jobs with access to the [[#System resources subjected to limitations|physical resources]] of Mufasa, such as CPUs, GPUs and RAM. Thanks to SLURM, processing jobs share system resources, optimising their occupation and availability. &lt;br /&gt;
&lt;br /&gt;
When a user runs a job, the job does not get executed immediately and is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039;. SLURM executes jobs according to their order in the queue: the top job in the queue gets executed as soon as the necessary resources are available, while jobs lower in the queue wait longer. The position of a job in the queue is due to the &amp;#039;&amp;#039;&amp;#039;[[#Job priority|priority]]&amp;#039;&amp;#039;&amp;#039; assigned to it by SLURM, with higher-priority jobs closer to the top. As a general rule:&lt;br /&gt;
&lt;br /&gt;
;: &amp;#039;&amp;#039;&amp;#039;the greater the fraction of Mufasa&amp;#039;s overall resources that a job asks for, the lower the job&amp;#039;s priority will be&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
The priority mechanism is used to encourage users to use Mufasa&amp;#039;s resources (i.e.: GPUs, CPUs, RAM, execution time) in an effective and equitable manner. This page includes a [[#How_to_maximise_the_priority_of_your_jobs|chart explaining how to maximise the priority of your jobs]].&lt;br /&gt;
&lt;br /&gt;
The &amp;#039;&amp;#039;&amp;#039;time&amp;#039;&amp;#039;&amp;#039; available to a job for its execution is controlled by SLURM. When a user requests execution of a job, they must specify the duration of the time slot that the job needs. The job must complete its execution before the end of the requested time slot, because at the end of such slot it gets killed by SLURM.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0 access to system resources is managed via SLURM&amp;#039;s [[#SLURM Quality of Service (QOS)|Quality of Service (QOS)]] mechanism (Mufasa 1.0 used [[#SLURM_partitions|partitions]] instead). While [[User Jobs#Running jobs with SLURM|launching a processing job via SLURM]], the user must always specify the chosen QOS. QOSes differ in the set of resources that they provide access to because each of them is designed to fit a given type of job.&lt;br /&gt;
&lt;br /&gt;
Mufasa sets limits to the number of jobs by the same user. This page includes a [[#Limits on jobs by the same user|table summarising such limits]].&lt;br /&gt;
&lt;br /&gt;
= SLURM Quality of Service (QOS) =&lt;br /&gt;
&lt;br /&gt;
Through &amp;#039;&amp;#039;&amp;#039;Quality of Services&amp;#039;&amp;#039;&amp;#039; (&amp;#039;&amp;#039;&amp;#039;QOSes&amp;#039;&amp;#039;&amp;#039;), SLURM lets system configurators assign a name to a set of related constraints.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, QOSes are used to define different levels of access to the server&amp;#039;s resources. When [[User Jobs|executing a job with SLURM]], a user must always &amp;#039;&amp;#039;&amp;#039;specify the QOS&amp;#039;&amp;#039;&amp;#039; that their job will use: this choice, in turn, determines what resources the job is able to access and influences the [[#Job priority|priority]] of the job.&lt;br /&gt;
&lt;br /&gt;
Mufasa&amp;#039;s QOSes and their features can be inspected with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list qos format=name%-11,priority,MaxSubmit,maxwall,maxtres%-80,mintres%-18&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following (see below for the same data in table form):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
Name          Priority MaxSubmit     MaxWall MaxTRES                                                                          MinTRES            &lt;br /&gt;
----------- ---------- --------- ----------- -------------------------------------------------------------------------------- ------------------ &lt;br /&gt;
normal               0                                                                                                                           &lt;br /&gt;
nogpu                4         1  3-00:00:00 cpu=16,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=128G                    &lt;br /&gt;
gpuheavy-20          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=2,mem=128G             gres/gpu:4g.20gb=1 &lt;br /&gt;
gpuheavy-40          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=1,gres/gpu:4g.20gb=0,mem=128G             gres/gpu:40gb=1    &lt;br /&gt;
gpulight             8         1    12:00:00 cpu=2,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpu                  2         1  1-00:00:00 cpu=8,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpuwide              2         2  1-00:00:00 cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=1,mem=64G              gres/gpu:4g.20gb=1 &lt;br /&gt;
build               32         1    02:00:00 cpu=2,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=16G&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The columns of this output are the following:&lt;br /&gt;
&lt;br /&gt;
:; Name&lt;br /&gt;
:: name of the QOS&lt;br /&gt;
&lt;br /&gt;
:; Priority&lt;br /&gt;
:: priority tier associated to the QOS (higher value = higher priority): see [[#Job priority|Job priority]] for details&lt;br /&gt;
&lt;br /&gt;
:; MaxSubmit&lt;br /&gt;
:: maximum number of jobs from a single user that can be submitted to SLURM with this QOS; submitted jobs include both running and queued jobs&lt;br /&gt;
:: See [[#Limits on jobs by the same user|Limits on jobs by the same user]] for an overview of the limits on jobs set by Mufasa.&lt;br /&gt;
&lt;br /&gt;
:; MaxWall&lt;br /&gt;
:: maximum wall clock duration of the jobs using the QOS (after which they are killed by SLURM), in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
:: For some QOSes these are not set: it means that they are determined by the [[#SLURM partitions|partition]]. Partitions also define the [[#Default values|default duration]] of jobs.&lt;br /&gt;
&lt;br /&gt;
:; MaxTRES&lt;br /&gt;
:: amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) available to a job using the QOS, where&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;cpu=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of CPUs (i.e., processor cores) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of CPUs specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;gres/&amp;#039;&amp;#039;gpu:Type&amp;#039;&amp;#039;=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of GPUs of class &amp;lt;code&amp;gt;&amp;#039;&amp;#039;Type&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; (see [[User Jobs#gres syntax|&amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax]]) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; (for QOSes that allow access to GPUs) if not specified, the job cannot be launched&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;mem=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;G&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum amount of system RAM is &amp;#039;&amp;#039;K&amp;#039;&amp;#039; GBytes&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of RAM specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
&lt;br /&gt;
:; MinTRES&lt;br /&gt;
:: minimum amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) that a job using the QOS must request in order to actually get executed by SLURM.&lt;br /&gt;
:: (If your job does not actually need these resources, you&amp;#039;ve chosen the wrong QOS: you can use one with a higher priority.)&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; QOS is the default one, and exists only to ensure that users always specify a QOS when running a job. Since &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; has zero priority and no resources, a job run using this QOS would never be run.&lt;br /&gt;
&lt;br /&gt;
The information provided by the &amp;lt;code&amp;gt;sacctmgr list qos&amp;lt;/code&amp;gt; command above is summarised by the following table:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! QOS&lt;br /&gt;
! Priority&amp;lt;/br&amp;gt;tier&lt;br /&gt;
! Max&amp;lt;/br&amp;gt;Submit&lt;br /&gt;
! MaxWall&amp;lt;/br&amp;gt;[h]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;CPUs&lt;br /&gt;
! max RAM&amp;lt;/br&amp;gt;[GB]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;3g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;4g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;40GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | build&lt;br /&gt;
| 32&lt;br /&gt;
| 1&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 16&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpulight&lt;br /&gt;
| 8&lt;br /&gt;
| 1&lt;br /&gt;
| 12&lt;br /&gt;
| 2&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | nogpu&lt;br /&gt;
| 4&lt;br /&gt;
| 1&lt;br /&gt;
| 72&lt;br /&gt;
| 16&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpu&lt;br /&gt;
| 2&lt;br /&gt;
| 1&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuwide&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-20&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| 2 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-40&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
A key piece of information in the table above is the &amp;#039;&amp;#039;&amp;#039;priority tier&amp;#039;&amp;#039;&amp;#039; associated to each QOS. In Mufasa 2.0, priority tiers are used to encourage users to use the &amp;#039;&amp;#039;&amp;#039;least powerful QOS that is compatible with their needs&amp;#039;&amp;#039;&amp;#039;, where &amp;quot;powerful&amp;quot; means &amp;quot;rich with resources&amp;quot;. Less powerful QOSes increase the priority of the jobs that use them, so these jobs tend to be executed sooner. See [[#Job priority|Job priority]] for details about how priority affects the execution order of jobs in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
== The &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS ==&lt;br /&gt;
&lt;br /&gt;
This QOS is specifically designed to be used by Mufasa users to &amp;#039;&amp;#039;&amp;#039;quickly build [[System#Containers|container images]]&amp;#039;&amp;#039;&amp;#039;. Its associated priority tier is very high: this, combined with the fact that jobs using this QOS require few resources, means that such jobs usually get executed very soon.&lt;br /&gt;
&lt;br /&gt;
On the other hand, the limited resources and the lack of access to the GPUs make the &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS unsuitable for other tasks.&lt;br /&gt;
&lt;br /&gt;
See [[Singularity#Building Singularity images|Building Singularity images]] for directions about building Singularity container images.&lt;br /&gt;
&lt;br /&gt;
== Restricted QOSes ==&lt;br /&gt;
&lt;br /&gt;
In Mufasa, the most powerful QOSes are reserved to researchers (called &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users: these include academic personnel and Ph.D. students). M.Sc. students, called &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users, cannot use them while running jobs.&lt;br /&gt;
&lt;br /&gt;
See [[#research users and students users|below]] to understand how &amp;#039;&amp;#039;user categories&amp;#039;&amp;#039; work in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
= &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users =&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa belong to two &amp;#039;&amp;#039;&amp;#039;user categories&amp;#039;&amp;#039;&amp;#039;. Categories provide users with different access to  to Mufasa&amp;#039;s resources: the idea being to provide researchers with more access while still letting students benefit from the server&amp;#039;s capabilities.&lt;br /&gt;
&lt;br /&gt;
User categories in Mufasa are the following:&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. academic personnel and Ph.D. students&lt;br /&gt;
::: * can use all QOSes, including the the [[#Restricted QOSes|restricted ones]]&lt;br /&gt;
::: * their jobs have a higher &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a higher number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. M.Sc. students&lt;br /&gt;
::: * cannot use the [[#Restricted QOSes|restricted QOSes]]&lt;br /&gt;
::: * their jobs have a lower &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a lower number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
You can inspect the differences in priority and running jobs between &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association format=&amp;quot;account,priority,maxjobs&amp;quot; | grep -E &amp;#039;Account|research|students&amp;#039;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
   Account   Priority MaxJobs &lt;br /&gt;
  research          4       2 &lt;br /&gt;
  students          1       1&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
To know what limits apply to your own user, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association where user=$USER format=&amp;quot;user,priority,maxjobs,qos%-60&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
      User   Priority MaxJobs QOS                                                          &lt;br /&gt;
---------- ---------- ------- ------------------------------------------------------------ &lt;br /&gt;
    preali          4       2 build,gpu,gpuheavy-20,gpuheavy-40,gpulight,gpuwide,nogpu&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The list under &amp;quot;QOS&amp;quot; shows what QOSes your user is allowed to use when [[User Jobs|running jobs]]. &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users can use all of them, while &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users can only access a subset of them.&lt;br /&gt;
&lt;br /&gt;
= Limits on jobs by the same user =&lt;br /&gt;
&lt;br /&gt;
Mufasa enforces limits on the number of jobs from a single user. Such limits aim at preventing users from &amp;quot;hogging&amp;quot; system resources, and apply to:&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;submitted jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that the user asked SLURM to execute, each of which may currently be either running or queued&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;running jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that are currently in execution&lt;br /&gt;
&lt;br /&gt;
Limits on submitted jobs are set via [[#SLURM Quality of Service (QOS)|QOSes]], while limits on running jobs are set via [[#research users and students users|user category]].&lt;br /&gt;
&lt;br /&gt;
The following table summarises what limits exist:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! &lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;running&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;submitted&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | global limits&amp;lt;br/&amp;gt;(system-wide)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to limits set by individual QOSes (below)&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | limits for&amp;lt;br/&amp;gt;individual&amp;lt;/br&amp;gt;QOSes&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to global limits (above)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for the&amp;#039;&amp;#039; &amp;lt;code&amp;gt;gpuwide&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;QOS&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for each of the other QOSes&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
= Job priority =&lt;br /&gt;
&lt;br /&gt;
Once the execution of a job has been requested, the job is not run immediately: it is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039; by SLURM, together with all the other jobs awaiting execution. The job on top of the queue at any time is the first to be put into execution as soon as the resources it requires are available. &lt;br /&gt;
&lt;br /&gt;
The order of the items in the job queue depends on their &amp;#039;&amp;#039;&amp;#039;priority&amp;#039;&amp;#039;&amp;#039;. The job with the highest priority is put by SLURM on top of the queue, while all other jobs are put in the queue in order of descending priority.&lt;br /&gt;
&lt;br /&gt;
The goal of SLURM is to maximise resource availability: i.e., to ensure the shorter possible wait time before job execution. To achieve this goal, in Mufasa SLURM is configured to &amp;#039;&amp;#039;&amp;#039;encourage users not to ask for resources or execution time that their job doesn&amp;#039;t need&amp;#039;&amp;#039;&amp;#039;. This is done via the priority mechanism: the more resources and/or time a job requests, the lower its priority will be; and the later it will be executed.&lt;br /&gt;
&lt;br /&gt;
Priority management in Mufasa is designed to set up a &amp;#039;&amp;#039;&amp;#039;virtuous cycle&amp;#039;&amp;#039;&amp;#039; where users, by carefully choosing what to ask for, obtain two results:&lt;br /&gt;
* they ensure that their job is executed as soon as possible;&lt;br /&gt;
* they leave as much as possible of Mufasa&amp;#039;s resources free for other users&amp;#039;s jobs.&lt;br /&gt;
&lt;br /&gt;
== Elements determining job priority ==&lt;br /&gt;
In Mufasa, the priority of a job is computed by SLURM according to the following elements:&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#research users and students users|User category]]&amp;#039;&amp;#039;&amp;#039; (i.e., &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;)&lt;br /&gt;
::: Used to provide higher priority to jobs run by &amp;#039;&amp;#039;&amp;#039;research personnel&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#SLURM Quality of Service (QOS)|QOS]]&amp;#039;&amp;#039;&amp;#039; used by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;less resources&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Number of CPUs&amp;#039;&amp;#039;&amp;#039; requested by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;a lower number of CPUs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job Age&amp;#039;&amp;#039;&amp;#039;, i.e. the time that the job has been waiting in the queue&lt;br /&gt;
::: Used to provide higher priority to jobs which have been &amp;#039;&amp;#039;&amp;#039;queued for a longer time&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;FairShare&amp;#039;&amp;#039;&amp;#039;, i.e. a factor computed by SLURM to balance use of the system by different users&lt;br /&gt;
::: Used to provide higher priority to jobs by users who &amp;#039;&amp;#039;&amp;#039;used Mufasa&amp;#039;s resources less than others&amp;#039;&amp;#039;&amp;#039; (see [[#How FairShare works|below]])&lt;br /&gt;
&lt;br /&gt;
You can understand how each of these elements contributes to the priority of the jobs waiting for execution (both yours and of other users) with command &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;. A tailored version of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; that only displays the parameters listed above is the following:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sprio -o &amp;quot;%6i %9u %.8Y %.10B %.5Q %.9T %.5A %.5F&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Here is an example of the output of the command above:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
JOBID  USER      PRIORITY      ASSOC   QOS      TRES   AGE FAIRS&lt;br /&gt;
5790   amoglia       3495       2500   375  cpu=-800   933   487&lt;br /&gt;
5791   qliu          3010       2500   375  cpu=-800   916    19&lt;br /&gt;
5817   mzoccola      2336        625   750  cpu=-800   495  1266&lt;br /&gt;
5822   fbottazzi     2235        625   750  cpu=-600   467   994&lt;br /&gt;
5837   lpassoni      2508       2500   750  cpu=-800     0    58&lt;br /&gt;
5838   bagostini     4437        625  3000  cpu=-200     0  1013&lt;br /&gt;
5839   flazzaro      1549        625   750  cpu=-800     0   974&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
In it, column PRIORITY provides the overall priority of each job. Each priority value is the sum of the five values to its right; in particular, column ASSOC corresponds to user category, while column TRES corresponds to the number of CPUs. Note that the contribution of TRES to priority is &amp;#039;&amp;#039;negative&amp;#039;&amp;#039;, i.e. it is a penalty; its value is proportional to the number of CPUs requested by the job.&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;Things to consider when using &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
* Its output is only relevant for jobs &amp;#039;&amp;#039;not yet in execution&amp;#039;&amp;#039;. For an already running job, the values shown by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; correspond to the priority that the job would get if evaluated now, which generally differs from its actual priority.&lt;br /&gt;
* Jobs listed by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; may temporarily differ from jobs listed by &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;: this is due to the fact that SLURM updates its internal data with a periodicity of a few minutes.&lt;br /&gt;
&lt;br /&gt;
=== How FairShare works ===&lt;br /&gt;
In Mufasa 2.0, the FairShare of each user is computed according to these rules:&lt;br /&gt;
# It considers the quantity used by (current and past) user jobs of:&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;CPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of CPUs used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;RAM&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the amount of RAM used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;GPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of GPUs used&lt;br /&gt;
#** note: 40 GB GPUs have double the impact on FairShare than 20 GB GPUs&lt;br /&gt;
# The impact on FairShare of any use of resources is &amp;#039;&amp;#039;&amp;#039;proportional to its duration&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
#* example: using 32 GB of RAM for 48h has the same impact of using 64 GB of RAM for 24h&lt;br /&gt;
# FairShare has a &amp;quot;fading memory&amp;quot;: i.e., resource use has less and less impact on FairShare the farther in the past it is&lt;br /&gt;
#* consequence: over time, the &amp;quot;history&amp;quot; of a user gets forgotten by FairShare&lt;br /&gt;
&lt;br /&gt;
== How to maximise the priority of your jobs ==&lt;br /&gt;
&lt;br /&gt;
Every time you run a SLURM job, follow these guidelines:&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Choose the less powerful QOS compatible with the needs of your job&lt;br /&gt;
:: QOSes with access to less resources have higher base priority&lt;br /&gt;
&lt;br /&gt;
; Only request CPUs and RAM that your job really needs&lt;br /&gt;
:: Usually, code exploits multiple CPUs only if specifically designed to do so: if unsure, only ask for 1 CPU&lt;br /&gt;
:: The fewer resources your job asks for, the less it will wait before they become available&lt;br /&gt;
:: The more CPUs your job asks for, the stronger the penalty to its priority&lt;br /&gt;
:: Asking for fewer CPUs and/or less RAM improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Do not request more time than your jobs needs to complete&lt;br /&gt;
:: Make a worst-case estimate and only ask for that duration&lt;br /&gt;
:: Asking for less of Mufasa&amp;#039;s time improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Debug and test code using less powerful QOSes before running it with more powerful QOSes&lt;br /&gt;
:: Your test jobs will get a higher priority and your FairShare will improve&lt;br /&gt;
&lt;br /&gt;
; Cancel jobs that you don&amp;#039;t need them anymore&lt;br /&gt;
:: Always use [[User_Jobs#Cancelling_a_job_with_scancel|scancel]] to delete completed (or crashed) jobs: your Fairshare will improve&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Protip: it&amp;#039;s a good idea to [[User Jobs#Looking for unused GPUs|check for unused GPUs]] before choosing what to request. Requesting a GPU that is currently idle will help your job get executed sooner.&lt;br /&gt;
&lt;br /&gt;
= SLURM partitions =&lt;br /&gt;
&lt;br /&gt;
Partitions are another mechanism provided by SLURM to create different levels of access to system resources. Since in Mufasa 2.0 access to resources is controlled via [[#SLURM Quality of Service (QOS)|QOSes]], partitions are not very relevant. (This is one of the main differences between Mufasa 1.0 and Mufasa 2.0.)&lt;br /&gt;
&lt;br /&gt;
Note, however, that the default values for some features of SLURM jobs (e.g., duration) are [[#Default values|set by the partition]].&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, there is a single SLURM partition, called &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt;, and all jobs run on it. The state of &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt; can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sinfo -o &amp;quot;%10P %5a %9T %11L %10l&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PARTITION  AVAIL STATE     DEFAULTTIME TIMELIMIT &lt;br /&gt;
jobs*      up    idle      1:00:00     3-00:00:00&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where columns correspond to the following information:&lt;br /&gt;
&lt;br /&gt;
:; PARTITION&lt;br /&gt;
:: name of the partition; the asterisks indicates that it&amp;#039;s the default one&lt;br /&gt;
&lt;br /&gt;
:; AVAIL&lt;br /&gt;
:: state/availability of the partition: see [[#Partition availability|below]]&lt;br /&gt;
&lt;br /&gt;
:; STATE&lt;br /&gt;
:: state (using [https://slurm.schedmd.com/sinfo.html#SECTION_NODE-STATE-CODES these codes])&lt;br /&gt;
:: typical values are &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;mixed&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that some of the resources are busy executing jobs while other are idle, and &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;allocated&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that all of the resources are in use&lt;br /&gt;
&lt;br /&gt;
:; DEFAULTTIME&lt;br /&gt;
:: default runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:; TIMELIMIT&lt;br /&gt;
:: maximum runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
The asterisk at the end of the partition name indicates the default partition, i.e. the one on which jobs which do not ask for a specific partition are run.&lt;br /&gt;
&lt;br /&gt;
Command &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; does not tell you about the &amp;#039;&amp;#039;jobs&amp;#039;&amp;#039; submitted to a partition. This information is obtained, instead, with [[User Jobs#Inspecting jobs with squeue|command &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;]].&lt;br /&gt;
&lt;br /&gt;
== Partition availability ==&lt;br /&gt;
&lt;br /&gt;
The most important information that &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; provides is the &amp;#039;&amp;#039;&amp;#039;availability&amp;#039;&amp;#039;&amp;#039; (also called &amp;#039;&amp;#039;&amp;#039;state&amp;#039;&amp;#039;&amp;#039;) of partitions. This is shown in column &amp;quot;AVAIL&amp;quot;. Possible partition states are:&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is available&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed as soon as resources allow&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is in the process of becoming unavailable (i.e., of entering the &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; state: see below)&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is unavailable&lt;br /&gt;
:: There are no running jobs&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
When a partition goes from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; no harm is done to running jobs. In a normally functioning SLURM system, the passage from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; happens only when no jobs are running on the partition. If (e.g., due to a malfunction) the passage happens with jobs still running, they get killed.&lt;br /&gt;
&lt;br /&gt;
A partition in state &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; requires intervention by a [[Roles|Job Administrator]] to be restored to &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
== Default values ==&lt;br /&gt;
&lt;br /&gt;
The features of SLURM partitions, including the &amp;#039;&amp;#039;&amp;#039;default values&amp;#039;&amp;#039;&amp;#039; which are applied to jobs that do not make explicit requests, can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scontrol show partition&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PartitionName=jobs&lt;br /&gt;
   AllowGroups=ALL AllowAccounts=ALL AllowQos=nogpu,gpulight,gpu,gpuwide,gpuheavy-20,gpuheavy-40&lt;br /&gt;
   AllocNodes=ALL Default=YES QoS=N/A&lt;br /&gt;
   DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO&lt;br /&gt;
   MaxNodes=UNLIMITED MaxTime=3-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED&lt;br /&gt;
   Nodes=gn01&lt;br /&gt;
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO&lt;br /&gt;
   OverTimeLimit=NONE PreemptMode=OFF&lt;br /&gt;
   State=UP TotalCPUs=48 TotalNodes=1 SelectTypeParameters=NONE&lt;br /&gt;
   JobDefaults=(null)&lt;br /&gt;
   DefMemPerNode=4096 MaxMemPerNode=UNLIMITED&lt;br /&gt;
   TRES=cpu=48,mem=1011435M,node=1,billing=49,gres/gpu=13,gres/gpu:3g.20gb=5,gres/gpu:40gb=3,gres/gpu:4g.20gb=5&lt;br /&gt;
   TRESBillingWeights=cpu=1.0,gres/gpu:3g.20gb=6.0,gres/gpu:4g.20gb=6.0,gres/gpu:40gb=6.0,mem=0.05g&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In the output of &amp;lt;code&amp;gt;scontrol show partition&amp;lt;/code&amp;gt; the most relevant default values for Mufasa users are:&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefaultTime&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default execution time assigned to a job run on the partition (e.g., 1 hour)&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefMemPerNode&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default amount of RAM assigned to a job run on the partition (e.g., 4GB)&lt;br /&gt;
&lt;br /&gt;
= System resources subjected to limitations =&lt;br /&gt;
&lt;br /&gt;
In systems based on SLURM like Mufasa, &amp;#039;&amp;#039;&amp;#039;TRES (Trackable RESources)&amp;#039;&amp;#039;&amp;#039; are (from [https://slurm.schedmd.com/tres.html SLURM&amp;#039;s documentation] &amp;quot;&amp;#039;&amp;#039;resources that can be tracked for usage or used to enforce limits against.&amp;#039;&amp;#039;&amp;quot;&lt;br /&gt;
&lt;br /&gt;
TRES include CPUs, RAM and &amp;#039;&amp;#039;&amp;#039;GRES&amp;#039;&amp;#039;&amp;#039;. The last term stands for &amp;#039;&amp;#039;Generic RESources&amp;#039;&amp;#039; that a job may need for its execution. In Mufasa, the only &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; resources are the GPUs.&lt;br /&gt;
&lt;br /&gt;
== &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax ==&lt;br /&gt;
&lt;br /&gt;
To ask SLURM to assign GRES resources (i.e., GPUs) to a job, a special syntax must be used. Precisely, the name of each GPU resource takes the form&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:Name:Type&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
Considering the [[System#CPUs and GPUs|GPU complement of Mufasa]], &amp;lt;code&amp;gt;Type&amp;lt;/code&amp;gt; takes the following values:&lt;br /&gt;
&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:40gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 40 Gbytes of RAM&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:4g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 4 compute units&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 3 compute units&lt;br /&gt;
&lt;br /&gt;
So, for instance,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
identifies a resource corresponding to a GPU with 20 GB of RAM and 3 compute units.&lt;br /&gt;
&lt;br /&gt;
When asking for a GRES resource (e.g., in an &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command or an &amp;lt;code&amp;gt;SBATCH&amp;lt;/code&amp;gt; directive of an [[User Jobs#Using execution scripts to run jobs|execution script]]), the syntax required by SLURM is&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:&amp;lt;Type&amp;gt;:&amp;lt;Quantity&amp;gt;&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
where &amp;lt;code&amp;gt;Quantity&amp;lt;/code&amp;gt; is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of type &amp;lt;code&amp;gt;4g.20gb&amp;lt;/code&amp;gt; the syntax is&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:4g.20gb:2&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SLURM&amp;#039;s &amp;#039;&amp;#039;generic resources&amp;#039;&amp;#039; are defined in &amp;lt;code&amp;gt;/etc/slurm/gres.conf&amp;lt;/code&amp;gt;. In order to make GPUs available to SLURM&amp;#039;s &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; management, Mufasa makes use of Nvidia&amp;#039;s [https://developer.nvidia.com/nvidia-management-library-nvml NVML library]. For additional information see [https://slurm.schedmd.com/gres.html SLURM&amp;#039;s documentation].&lt;/div&gt;</summary>
		<author><name>GiulioFontana</name></author>
	</entry>
	<entry>
		<id>https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2591</id>
		<title>SLURM</title>
		<link rel="alternate" type="text/html" href="https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2591"/>
		<updated>2026-11-04T09:12:33Z</updated>

		<summary type="html">&lt;p&gt;GiulioFontana: /* Elements determining job priority */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page presents the features of SLURM that are most relevant to Mufasa&amp;#039;s [[Roles|Job Users]]. Job Users can submit jobs for execution, cancel their own jobs, and see other users&amp;#039; jobs (but not intervene on them).&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa &amp;#039;&amp;#039;&amp;#039;must use SLURM&amp;#039;&amp;#039;&amp;#039; to run resource-heavy processes, i.e. computing jobs that require one or more of the following:&lt;br /&gt;
* GPUs&lt;br /&gt;
* multiple CPUs&lt;br /&gt;
* powerful CPUs&lt;br /&gt;
* a significant amount of RAM&lt;br /&gt;
&lt;br /&gt;
In fact, only processes run via SLURM have access to all the resources of Mufasa. Processes run outside SLURM are executed by the [[System#Login server|login server]] virtual machine, which has minimal resources and no GPUs. Using SLURM is therefore the only way to execute resource-heavy jobs on Mufasa (this is a key difference between Mufasa 1.0 and Mufasa 2.0).&lt;br /&gt;
&lt;br /&gt;
= SLURM in a nutshell =&lt;br /&gt;
&lt;br /&gt;
Computation jobs on Mufasa needs to be launched via [[System#The SLURM job scheduling system|SLURM]]. SLURM provides jobs with access to the [[#System resources subjected to limitations|physical resources]] of Mufasa, such as CPUs, GPUs and RAM. Thanks to SLURM, processing jobs share system resources, optimising their occupation and availability. &lt;br /&gt;
&lt;br /&gt;
When a user runs a job, the job does not get executed immediately and is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039;. SLURM executes jobs according to their order in the queue: the top job in the queue gets executed as soon as the necessary resources are available, while jobs lower in the queue wait longer. The position of a job in the queue is due to the &amp;#039;&amp;#039;&amp;#039;[[#Job priority|priority]]&amp;#039;&amp;#039;&amp;#039; assigned to it by SLURM, with higher-priority jobs closer to the top. As a general rule:&lt;br /&gt;
&lt;br /&gt;
;: &amp;#039;&amp;#039;&amp;#039;the greater the fraction of Mufasa&amp;#039;s overall resources that a job asks for, the lower the job&amp;#039;s priority will be&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
The priority mechanism is used to encourage users to use Mufasa&amp;#039;s resources (i.e.: GPUs, CPUs, RAM, execution time) in an effective and equitable manner. This page includes a [[#How_to_maximise_the_priority_of_your_jobs|chart explaining how to maximise the priority of your jobs]].&lt;br /&gt;
&lt;br /&gt;
The &amp;#039;&amp;#039;&amp;#039;time&amp;#039;&amp;#039;&amp;#039; available to a job for its execution is controlled by SLURM. When a user requests execution of a job, they must specify the duration of the time slot that the job needs. The job must complete its execution before the end of the requested time slot, because at the end of such slot it gets killed by SLURM.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0 access to system resources is managed via SLURM&amp;#039;s [[#SLURM Quality of Service (QOS)|Quality of Service (QOS)]] mechanism (Mufasa 1.0 used [[#SLURM_partitions|partitions]] instead). While [[User Jobs#Running jobs with SLURM|launching a processing job via SLURM]], the user must always specify the chosen QOS. QOSes differ in the set of resources that they provide access to because each of them is designed to fit a given type of job.&lt;br /&gt;
&lt;br /&gt;
Mufasa sets limits to the number of jobs by the same user. This page includes a [[#Limits on jobs by the same user|table summarising such limits]].&lt;br /&gt;
&lt;br /&gt;
= SLURM Quality of Service (QOS) =&lt;br /&gt;
&lt;br /&gt;
Through &amp;#039;&amp;#039;&amp;#039;Quality of Services&amp;#039;&amp;#039;&amp;#039; (&amp;#039;&amp;#039;&amp;#039;QOSes&amp;#039;&amp;#039;&amp;#039;), SLURM lets system configurators assign a name to a set of related constraints.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, QOSes are used to define different levels of access to the server&amp;#039;s resources. When [[User Jobs|executing a job with SLURM]], a user must always &amp;#039;&amp;#039;&amp;#039;specify the QOS&amp;#039;&amp;#039;&amp;#039; that their job will use: this choice, in turn, determines what resources the job is able to access and influences the [[#Job priority|priority]] of the job.&lt;br /&gt;
&lt;br /&gt;
Mufasa&amp;#039;s QOSes and their features can be inspected with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list qos format=name%-11,priority,MaxSubmit,maxwall,maxtres%-80,mintres%-18&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following (see below for the same data in table form):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
Name          Priority MaxSubmit     MaxWall MaxTRES                                                                          MinTRES            &lt;br /&gt;
----------- ---------- --------- ----------- -------------------------------------------------------------------------------- ------------------ &lt;br /&gt;
normal               0                                                                                                                           &lt;br /&gt;
nogpu                4         1  3-00:00:00 cpu=16,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=128G                    &lt;br /&gt;
gpuheavy-20          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=2,mem=128G             gres/gpu:4g.20gb=1 &lt;br /&gt;
gpuheavy-40          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=1,gres/gpu:4g.20gb=0,mem=128G             gres/gpu:40gb=1    &lt;br /&gt;
gpulight             8         1    12:00:00 cpu=2,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpu                  2         1  1-00:00:00 cpu=8,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpuwide              2         2  1-00:00:00 cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=1,mem=64G              gres/gpu:4g.20gb=1 &lt;br /&gt;
build               32         1    02:00:00 cpu=2,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=16G&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The columns of this output are the following:&lt;br /&gt;
&lt;br /&gt;
:; Name&lt;br /&gt;
:: name of the QOS&lt;br /&gt;
&lt;br /&gt;
:; Priority&lt;br /&gt;
:: priority tier associated to the QOS (higher value = higher priority): see [[#Job priority|Job priority]] for details&lt;br /&gt;
&lt;br /&gt;
:; MaxSubmit&lt;br /&gt;
:: maximum number of jobs from a single user that can be submitted to SLURM with this QOS; submitted jobs include both running and queued jobs&lt;br /&gt;
:: See [[#Limits on jobs by the same user|Limits on jobs by the same user]] for an overview of the limits on jobs set by Mufasa.&lt;br /&gt;
&lt;br /&gt;
:; MaxWall&lt;br /&gt;
:: maximum wall clock duration of the jobs using the QOS (after which they are killed by SLURM), in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
:: For some QOSes these are not set: it means that they are determined by the [[#SLURM partitions|partition]]. Partitions also define the [[#Default values|default duration]] of jobs.&lt;br /&gt;
&lt;br /&gt;
:; MaxTRES&lt;br /&gt;
:: amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) available to a job using the QOS, where&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;cpu=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of CPUs (i.e., processor cores) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of CPUs specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;gres/&amp;#039;&amp;#039;gpu:Type&amp;#039;&amp;#039;=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of GPUs of class &amp;lt;code&amp;gt;&amp;#039;&amp;#039;Type&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; (see [[User Jobs#gres syntax|&amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax]]) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; (for QOSes that allow access to GPUs) if not specified, the job cannot be launched&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;mem=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;G&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum amount of system RAM is &amp;#039;&amp;#039;K&amp;#039;&amp;#039; GBytes&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of RAM specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
&lt;br /&gt;
:; MinTRES&lt;br /&gt;
:: minimum amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) that a job using the QOS must request in order to actually get executed by SLURM.&lt;br /&gt;
:: (If your job does not actually need these resources, you&amp;#039;ve chosen the wrong QOS: you can use one with a higher priority.)&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; QOS is the default one, and exists only to ensure that users always specify a QOS when running a job. Since &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; has zero priority and no resources, a job run using this QOS would never be run.&lt;br /&gt;
&lt;br /&gt;
The information provided by the &amp;lt;code&amp;gt;sacctmgr list qos&amp;lt;/code&amp;gt; command above is summarised by the following table:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! QOS&lt;br /&gt;
! Priority&amp;lt;/br&amp;gt;tier&lt;br /&gt;
! Max&amp;lt;/br&amp;gt;Submit&lt;br /&gt;
! MaxWall&amp;lt;/br&amp;gt;[h]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;CPUs&lt;br /&gt;
! max RAM&amp;lt;/br&amp;gt;[GB]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;3g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;4g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;40GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | build&lt;br /&gt;
| 32&lt;br /&gt;
| 1&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 16&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpulight&lt;br /&gt;
| 8&lt;br /&gt;
| 1&lt;br /&gt;
| 12&lt;br /&gt;
| 2&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | nogpu&lt;br /&gt;
| 4&lt;br /&gt;
| 1&lt;br /&gt;
| 72&lt;br /&gt;
| 16&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpu&lt;br /&gt;
| 2&lt;br /&gt;
| 1&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuwide&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-20&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| 2 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-40&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
A key piece of information in the table above is the &amp;#039;&amp;#039;&amp;#039;priority tier&amp;#039;&amp;#039;&amp;#039; associated to each QOS. In Mufasa 2.0, priority tiers are used to encourage users to use the &amp;#039;&amp;#039;&amp;#039;least powerful QOS that is compatible with their needs&amp;#039;&amp;#039;&amp;#039;, where &amp;quot;powerful&amp;quot; means &amp;quot;rich with resources&amp;quot;. Less powerful QOSes increase the priority of the jobs that use them, so these jobs tend to be executed sooner. See [[#Job priority|Job priority]] for details about how priority affects the execution order of jobs in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
== The &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS ==&lt;br /&gt;
&lt;br /&gt;
This QOS is specifically designed to be used by Mufasa users to &amp;#039;&amp;#039;&amp;#039;quickly build [[System#Containers|container images]]&amp;#039;&amp;#039;&amp;#039;. Its associated priority tier is very high: this, combined with the fact that jobs using this QOS require few resources, means that such jobs usually get executed very soon.&lt;br /&gt;
&lt;br /&gt;
On the other hand, the limited resources and the lack of access to the GPUs make the &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS unsuitable for other tasks.&lt;br /&gt;
&lt;br /&gt;
See [[Singularity#Building Singularity images|Building Singularity images]] for directions about building Singularity container images.&lt;br /&gt;
&lt;br /&gt;
== Restricted QOSes ==&lt;br /&gt;
&lt;br /&gt;
In Mufasa, the most powerful QOSes are reserved to researchers (called &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users: these include academic personnel and Ph.D. students). M.Sc. students, called &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users, cannot use them while running jobs.&lt;br /&gt;
&lt;br /&gt;
See [[#research users and students users|below]] to understand how &amp;#039;&amp;#039;user categories&amp;#039;&amp;#039; work in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
= &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users =&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa belong to two &amp;#039;&amp;#039;&amp;#039;user categories&amp;#039;&amp;#039;&amp;#039;. Categories provide users with different access to  to Mufasa&amp;#039;s resources: the idea being to provide researchers with more access while still letting students benefit from the server&amp;#039;s capabilities.&lt;br /&gt;
&lt;br /&gt;
User categories in Mufasa are the following:&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. academic personnel and Ph.D. students&lt;br /&gt;
::: * can use all QOSes, including the the [[#Restricted QOSes|restricted ones]]&lt;br /&gt;
::: * their jobs have a higher &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a higher number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. M.Sc. students&lt;br /&gt;
::: * cannot use the [[#Restricted QOSes|restricted QOSes]]&lt;br /&gt;
::: * their jobs have a lower &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a lower number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
You can inspect the differences in priority and running jobs between &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association format=&amp;quot;account,priority,maxjobs&amp;quot; | grep -E &amp;#039;Account|research|students&amp;#039;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
   Account   Priority MaxJobs &lt;br /&gt;
  research          4       2 &lt;br /&gt;
  students          1       1&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
To know what limits apply to your own user, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association where user=$USER format=&amp;quot;user,priority,maxjobs,qos%-60&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
      User   Priority MaxJobs QOS                                                          &lt;br /&gt;
---------- ---------- ------- ------------------------------------------------------------ &lt;br /&gt;
    preali          4       2 build,gpu,gpuheavy-20,gpuheavy-40,gpulight,gpuwide,nogpu&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The list under &amp;quot;QOS&amp;quot; shows what QOSes your user is allowed to use when [[User Jobs|running jobs]]. &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users can use all of them, while &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users can only access a subset of them.&lt;br /&gt;
&lt;br /&gt;
= Limits on jobs by the same user =&lt;br /&gt;
&lt;br /&gt;
Mufasa enforces limits on the number of jobs from a single user. Such limits aim at preventing users from &amp;quot;hogging&amp;quot; system resources, and apply to:&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;submitted jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that the user asked SLURM to execute, each of which may currently be either running or queued&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;running jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that are currently in execution&lt;br /&gt;
&lt;br /&gt;
Limits on submitted jobs are set via [[#SLURM Quality of Service (QOS)|QOSes]], while limits on running jobs are set via [[#research users and students users|user category]].&lt;br /&gt;
&lt;br /&gt;
The following table summarises what limits exist:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! &lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;running&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;submitted&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | global limits&amp;lt;br/&amp;gt;(system-wide)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to limits set by individual QOSes (below)&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | limits for&amp;lt;br/&amp;gt;individual&amp;lt;/br&amp;gt;QOSes&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to global limits (above)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for the&amp;#039;&amp;#039; &amp;lt;code&amp;gt;gpuwide&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;QOS&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for each of the other QOSes&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
= Job priority =&lt;br /&gt;
&lt;br /&gt;
Once the execution of a job has been requested, the job is not run immediately: it is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039; by SLURM, together with all the other jobs awaiting execution. The job on top of the queue at any time is the first to be put into execution as soon as the resources it requires are available. &lt;br /&gt;
&lt;br /&gt;
The order of the items in the job queue depends on their &amp;#039;&amp;#039;&amp;#039;priority&amp;#039;&amp;#039;&amp;#039;. The job with the highest priority is put by SLURM on top of the queue, while all other jobs are put in the queue in order of descending priority.&lt;br /&gt;
&lt;br /&gt;
The goal of SLURM is to maximise resource availability: i.e., to ensure the shorter possible wait time before job execution. To achieve this goal, in Mufasa SLURM is configured to &amp;#039;&amp;#039;&amp;#039;encourage users not to ask for resources or execution time that their job doesn&amp;#039;t need&amp;#039;&amp;#039;&amp;#039;. This is done via the priority mechanism: the more resources and/or time a job requests, the lower its priority will be; and the later it will be executed.&lt;br /&gt;
&lt;br /&gt;
Priority management in Mufasa is designed to set up a &amp;#039;&amp;#039;&amp;#039;virtuous cycle&amp;#039;&amp;#039;&amp;#039; where users, by carefully choosing what to ask for, obtain two results:&lt;br /&gt;
* they ensure that their job is executed as soon as possible;&lt;br /&gt;
* they leave as much as possible of Mufasa&amp;#039;s resources free for other users&amp;#039;s jobs.&lt;br /&gt;
&lt;br /&gt;
== Elements determining job priority ==&lt;br /&gt;
In Mufasa, the priority of a job is computed by SLURM according to the following elements:&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#research users and students users|User category]]&amp;#039;&amp;#039;&amp;#039; (i.e., &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;)&lt;br /&gt;
::: Used to provide higher priority to jobs run by &amp;#039;&amp;#039;&amp;#039;research personnel&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#SLURM Quality of Service (QOS)|QOS]]&amp;#039;&amp;#039;&amp;#039; used by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;less resources&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Number of CPUs&amp;#039;&amp;#039;&amp;#039; requested by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;a lower number of CPUs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job Age&amp;#039;&amp;#039;&amp;#039;, i.e. the time that the job has been waiting in the queue&lt;br /&gt;
::: Used to provide higher priority to jobs which have been &amp;#039;&amp;#039;&amp;#039;queued for a longer time&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;FairShare&amp;#039;&amp;#039;&amp;#039;, i.e. a factor computed by SLURM to balance use of the system by different users&lt;br /&gt;
::: Used to provide higher priority to jobs by users who &amp;#039;&amp;#039;&amp;#039;used Mufasa&amp;#039;s resources less than others&amp;#039;&amp;#039;&amp;#039; (see [[#How FairShare works|below]])&lt;br /&gt;
&lt;br /&gt;
You can understand how each of these elements contributes to the priority of the jobs waiting for execution (both yours and of other users) with command &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;. A tailored version of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; that only displays the parameters listed above is the following:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sprio -o &amp;quot;%6i %9u %.8Y %.10B %.5Q %.9T %.5A %.5F&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Here is an example of the output of the command above:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
JOBID  USER      PRIORITY      ASSOC   QOS      TRES   AGE FAIRS&lt;br /&gt;
5774   itombini      3576       2500   750  cpu=-800   990   136&lt;br /&gt;
5780   fbottazzi     2350        625   750  cpu=-600   504  1071&lt;br /&gt;
5783   mzoccola      1849        625   750  cpu=-800   300   974&lt;br /&gt;
5790   amoglia       2801       2500   375  cpu=-800   239   487&lt;br /&gt;
5791   qliu          2315       2500   375  cpu=-800   221    19&lt;br /&gt;
5795   lpassoni      2600       2500   750  cpu=-800    92    58&lt;br /&gt;
5805   adurat        3154       2500   750  cpu=-800    62   643&lt;br /&gt;
5806   itombini      2645       2500   750  cpu=-800    59   136&lt;br /&gt;
5813   fpastori      5385       2500  3000  cpu=-200    47    39&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
In it, column PRIORITY provides the overall priority of each job. Each priority value is the sum of the five values to its right; in particular, column ASSOC corresponds to user category, while column TRES corresponds to the number of CPUs. Note that the contribution of TRES to priority is &amp;#039;&amp;#039;negative&amp;#039;&amp;#039;, i.e. it is a penalty; its value is proportional to the number of CPUs requested by the job.&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;Things to consider when using &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
* Its output is only relevant for jobs &amp;#039;&amp;#039;not yet in execution&amp;#039;&amp;#039;. For an already running job, the values shown by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; correspond to the priority that the job would get if evaluated now, which generally differs from its actual priority.&lt;br /&gt;
* Jobs listed by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; may temporarily differ from jobs listed by &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;: this is due to the fact that SLURM updates its internal data with a periodicity of a few minutes.&lt;br /&gt;
&lt;br /&gt;
=== How FairShare works ===&lt;br /&gt;
In Mufasa 2.0, the FairShare of each user is computed according to these rules:&lt;br /&gt;
# It considers the quantity used by (current and past) user jobs of:&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;CPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of CPUs used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;RAM&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the amount of RAM used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;GPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of GPUs used&lt;br /&gt;
#** note: 40 GB GPUs have double the impact on FairShare than 20 GB GPUs&lt;br /&gt;
# The impact on FairShare of any use of resources is &amp;#039;&amp;#039;&amp;#039;proportional to its duration&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
#* example: using 32 GB of RAM for 48h has the same impact of using 64 GB of RAM for 24h&lt;br /&gt;
# FairShare has a &amp;quot;fading memory&amp;quot;: i.e., resource use has less and less impact on FairShare the farther in the past it is&lt;br /&gt;
#* consequence: over time, the &amp;quot;history&amp;quot; of a user gets forgotten by FairShare&lt;br /&gt;
&lt;br /&gt;
== How to maximise the priority of your jobs ==&lt;br /&gt;
&lt;br /&gt;
Every time you run a SLURM job, follow these guidelines:&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Choose the less powerful QOS compatible with the needs of your job&lt;br /&gt;
:: QOSes with access to less resources have higher base priority&lt;br /&gt;
&lt;br /&gt;
; Only request CPUs and RAM that your job really needs&lt;br /&gt;
:: Usually, code exploits multiple CPUs only if specifically designed to do so: if unsure, only ask for 1 CPU&lt;br /&gt;
:: The fewer resources your job asks for, the less it will wait before they become available&lt;br /&gt;
:: The more CPUs your job asks for, the stronger the penalty to its priority&lt;br /&gt;
:: Asking for fewer CPUs and/or less RAM improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Do not request more time than your jobs needs to complete&lt;br /&gt;
:: Make a worst-case estimate and only ask for that duration&lt;br /&gt;
:: Asking for less of Mufasa&amp;#039;s time improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Debug and test code using less powerful QOSes before running it with more powerful QOSes&lt;br /&gt;
:: Your test jobs will get a higher priority and your FairShare will improve&lt;br /&gt;
&lt;br /&gt;
; Cancel jobs that you don&amp;#039;t need them anymore&lt;br /&gt;
:: Always use [[User_Jobs#Cancelling_a_job_with_scancel|scancel]] to delete completed (or crashed) jobs: your Fairshare will improve&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Protip: it&amp;#039;s a good idea to [[User Jobs#Looking for unused GPUs|check for unused GPUs]] before choosing what to request. Requesting a GPU that is currently idle will help your job get executed sooner.&lt;br /&gt;
&lt;br /&gt;
= SLURM partitions =&lt;br /&gt;
&lt;br /&gt;
Partitions are another mechanism provided by SLURM to create different levels of access to system resources. Since in Mufasa 2.0 access to resources is controlled via [[#SLURM Quality of Service (QOS)|QOSes]], partitions are not very relevant. (This is one of the main differences between Mufasa 1.0 and Mufasa 2.0.)&lt;br /&gt;
&lt;br /&gt;
Note, however, that the default values for some features of SLURM jobs (e.g., duration) are [[#Default values|set by the partition]].&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, there is a single SLURM partition, called &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt;, and all jobs run on it. The state of &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt; can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sinfo -o &amp;quot;%10P %5a %9T %11L %10l&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PARTITION  AVAIL STATE     DEFAULTTIME TIMELIMIT &lt;br /&gt;
jobs*      up    idle      1:00:00     3-00:00:00&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where columns correspond to the following information:&lt;br /&gt;
&lt;br /&gt;
:; PARTITION&lt;br /&gt;
:: name of the partition; the asterisks indicates that it&amp;#039;s the default one&lt;br /&gt;
&lt;br /&gt;
:; AVAIL&lt;br /&gt;
:: state/availability of the partition: see [[#Partition availability|below]]&lt;br /&gt;
&lt;br /&gt;
:; STATE&lt;br /&gt;
:: state (using [https://slurm.schedmd.com/sinfo.html#SECTION_NODE-STATE-CODES these codes])&lt;br /&gt;
:: typical values are &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;mixed&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that some of the resources are busy executing jobs while other are idle, and &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;allocated&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that all of the resources are in use&lt;br /&gt;
&lt;br /&gt;
:; DEFAULTTIME&lt;br /&gt;
:: default runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:; TIMELIMIT&lt;br /&gt;
:: maximum runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
The asterisk at the end of the partition name indicates the default partition, i.e. the one on which jobs which do not ask for a specific partition are run.&lt;br /&gt;
&lt;br /&gt;
Command &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; does not tell you about the &amp;#039;&amp;#039;jobs&amp;#039;&amp;#039; submitted to a partition. This information is obtained, instead, with [[User Jobs#Inspecting jobs with squeue|command &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;]].&lt;br /&gt;
&lt;br /&gt;
== Partition availability ==&lt;br /&gt;
&lt;br /&gt;
The most important information that &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; provides is the &amp;#039;&amp;#039;&amp;#039;availability&amp;#039;&amp;#039;&amp;#039; (also called &amp;#039;&amp;#039;&amp;#039;state&amp;#039;&amp;#039;&amp;#039;) of partitions. This is shown in column &amp;quot;AVAIL&amp;quot;. Possible partition states are:&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is available&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed as soon as resources allow&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is in the process of becoming unavailable (i.e., of entering the &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; state: see below)&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is unavailable&lt;br /&gt;
:: There are no running jobs&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
When a partition goes from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; no harm is done to running jobs. In a normally functioning SLURM system, the passage from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; happens only when no jobs are running on the partition. If (e.g., due to a malfunction) the passage happens with jobs still running, they get killed.&lt;br /&gt;
&lt;br /&gt;
A partition in state &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; requires intervention by a [[Roles|Job Administrator]] to be restored to &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
== Default values ==&lt;br /&gt;
&lt;br /&gt;
The features of SLURM partitions, including the &amp;#039;&amp;#039;&amp;#039;default values&amp;#039;&amp;#039;&amp;#039; which are applied to jobs that do not make explicit requests, can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scontrol show partition&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PartitionName=jobs&lt;br /&gt;
   AllowGroups=ALL AllowAccounts=ALL AllowQos=nogpu,gpulight,gpu,gpuwide,gpuheavy-20,gpuheavy-40&lt;br /&gt;
   AllocNodes=ALL Default=YES QoS=N/A&lt;br /&gt;
   DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO&lt;br /&gt;
   MaxNodes=UNLIMITED MaxTime=3-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED&lt;br /&gt;
   Nodes=gn01&lt;br /&gt;
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO&lt;br /&gt;
   OverTimeLimit=NONE PreemptMode=OFF&lt;br /&gt;
   State=UP TotalCPUs=48 TotalNodes=1 SelectTypeParameters=NONE&lt;br /&gt;
   JobDefaults=(null)&lt;br /&gt;
   DefMemPerNode=4096 MaxMemPerNode=UNLIMITED&lt;br /&gt;
   TRES=cpu=48,mem=1011435M,node=1,billing=49,gres/gpu=13,gres/gpu:3g.20gb=5,gres/gpu:40gb=3,gres/gpu:4g.20gb=5&lt;br /&gt;
   TRESBillingWeights=cpu=1.0,gres/gpu:3g.20gb=6.0,gres/gpu:4g.20gb=6.0,gres/gpu:40gb=6.0,mem=0.05g&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In the output of &amp;lt;code&amp;gt;scontrol show partition&amp;lt;/code&amp;gt; the most relevant default values for Mufasa users are:&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefaultTime&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default execution time assigned to a job run on the partition (e.g., 1 hour)&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefMemPerNode&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default amount of RAM assigned to a job run on the partition (e.g., 4GB)&lt;br /&gt;
&lt;br /&gt;
= System resources subjected to limitations =&lt;br /&gt;
&lt;br /&gt;
In systems based on SLURM like Mufasa, &amp;#039;&amp;#039;&amp;#039;TRES (Trackable RESources)&amp;#039;&amp;#039;&amp;#039; are (from [https://slurm.schedmd.com/tres.html SLURM&amp;#039;s documentation] &amp;quot;&amp;#039;&amp;#039;resources that can be tracked for usage or used to enforce limits against.&amp;#039;&amp;#039;&amp;quot;&lt;br /&gt;
&lt;br /&gt;
TRES include CPUs, RAM and &amp;#039;&amp;#039;&amp;#039;GRES&amp;#039;&amp;#039;&amp;#039;. The last term stands for &amp;#039;&amp;#039;Generic RESources&amp;#039;&amp;#039; that a job may need for its execution. In Mufasa, the only &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; resources are the GPUs.&lt;br /&gt;
&lt;br /&gt;
== &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax ==&lt;br /&gt;
&lt;br /&gt;
To ask SLURM to assign GRES resources (i.e., GPUs) to a job, a special syntax must be used. Precisely, the name of each GPU resource takes the form&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:Name:Type&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
Considering the [[System#CPUs and GPUs|GPU complement of Mufasa]], &amp;lt;code&amp;gt;Type&amp;lt;/code&amp;gt; takes the following values:&lt;br /&gt;
&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:40gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 40 Gbytes of RAM&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:4g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 4 compute units&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 3 compute units&lt;br /&gt;
&lt;br /&gt;
So, for instance,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
identifies a resource corresponding to a GPU with 20 GB of RAM and 3 compute units.&lt;br /&gt;
&lt;br /&gt;
When asking for a GRES resource (e.g., in an &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command or an &amp;lt;code&amp;gt;SBATCH&amp;lt;/code&amp;gt; directive of an [[User Jobs#Using execution scripts to run jobs|execution script]]), the syntax required by SLURM is&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:&amp;lt;Type&amp;gt;:&amp;lt;Quantity&amp;gt;&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
where &amp;lt;code&amp;gt;Quantity&amp;lt;/code&amp;gt; is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of type &amp;lt;code&amp;gt;4g.20gb&amp;lt;/code&amp;gt; the syntax is&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:4g.20gb:2&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SLURM&amp;#039;s &amp;#039;&amp;#039;generic resources&amp;#039;&amp;#039; are defined in &amp;lt;code&amp;gt;/etc/slurm/gres.conf&amp;lt;/code&amp;gt;. In order to make GPUs available to SLURM&amp;#039;s &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; management, Mufasa makes use of Nvidia&amp;#039;s [https://developer.nvidia.com/nvidia-management-library-nvml NVML library]. For additional information see [https://slurm.schedmd.com/gres.html SLURM&amp;#039;s documentation].&lt;/div&gt;</summary>
		<author><name>GiulioFontana</name></author>
	</entry>
	<entry>
		<id>https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2590</id>
		<title>SLURM</title>
		<link rel="alternate" type="text/html" href="https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2590"/>
		<updated>2026-11-04T09:11:11Z</updated>

		<summary type="html">&lt;p&gt;GiulioFontana: /* Elements determining job priority */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page presents the features of SLURM that are most relevant to Mufasa&amp;#039;s [[Roles|Job Users]]. Job Users can submit jobs for execution, cancel their own jobs, and see other users&amp;#039; jobs (but not intervene on them).&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa &amp;#039;&amp;#039;&amp;#039;must use SLURM&amp;#039;&amp;#039;&amp;#039; to run resource-heavy processes, i.e. computing jobs that require one or more of the following:&lt;br /&gt;
* GPUs&lt;br /&gt;
* multiple CPUs&lt;br /&gt;
* powerful CPUs&lt;br /&gt;
* a significant amount of RAM&lt;br /&gt;
&lt;br /&gt;
In fact, only processes run via SLURM have access to all the resources of Mufasa. Processes run outside SLURM are executed by the [[System#Login server|login server]] virtual machine, which has minimal resources and no GPUs. Using SLURM is therefore the only way to execute resource-heavy jobs on Mufasa (this is a key difference between Mufasa 1.0 and Mufasa 2.0).&lt;br /&gt;
&lt;br /&gt;
= SLURM in a nutshell =&lt;br /&gt;
&lt;br /&gt;
Computation jobs on Mufasa needs to be launched via [[System#The SLURM job scheduling system|SLURM]]. SLURM provides jobs with access to the [[#System resources subjected to limitations|physical resources]] of Mufasa, such as CPUs, GPUs and RAM. Thanks to SLURM, processing jobs share system resources, optimising their occupation and availability. &lt;br /&gt;
&lt;br /&gt;
When a user runs a job, the job does not get executed immediately and is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039;. SLURM executes jobs according to their order in the queue: the top job in the queue gets executed as soon as the necessary resources are available, while jobs lower in the queue wait longer. The position of a job in the queue is due to the &amp;#039;&amp;#039;&amp;#039;[[#Job priority|priority]]&amp;#039;&amp;#039;&amp;#039; assigned to it by SLURM, with higher-priority jobs closer to the top. As a general rule:&lt;br /&gt;
&lt;br /&gt;
;: &amp;#039;&amp;#039;&amp;#039;the greater the fraction of Mufasa&amp;#039;s overall resources that a job asks for, the lower the job&amp;#039;s priority will be&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
The priority mechanism is used to encourage users to use Mufasa&amp;#039;s resources (i.e.: GPUs, CPUs, RAM, execution time) in an effective and equitable manner. This page includes a [[#How_to_maximise_the_priority_of_your_jobs|chart explaining how to maximise the priority of your jobs]].&lt;br /&gt;
&lt;br /&gt;
The &amp;#039;&amp;#039;&amp;#039;time&amp;#039;&amp;#039;&amp;#039; available to a job for its execution is controlled by SLURM. When a user requests execution of a job, they must specify the duration of the time slot that the job needs. The job must complete its execution before the end of the requested time slot, because at the end of such slot it gets killed by SLURM.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0 access to system resources is managed via SLURM&amp;#039;s [[#SLURM Quality of Service (QOS)|Quality of Service (QOS)]] mechanism (Mufasa 1.0 used [[#SLURM_partitions|partitions]] instead). While [[User Jobs#Running jobs with SLURM|launching a processing job via SLURM]], the user must always specify the chosen QOS. QOSes differ in the set of resources that they provide access to because each of them is designed to fit a given type of job.&lt;br /&gt;
&lt;br /&gt;
Mufasa sets limits to the number of jobs by the same user. This page includes a [[#Limits on jobs by the same user|table summarising such limits]].&lt;br /&gt;
&lt;br /&gt;
= SLURM Quality of Service (QOS) =&lt;br /&gt;
&lt;br /&gt;
Through &amp;#039;&amp;#039;&amp;#039;Quality of Services&amp;#039;&amp;#039;&amp;#039; (&amp;#039;&amp;#039;&amp;#039;QOSes&amp;#039;&amp;#039;&amp;#039;), SLURM lets system configurators assign a name to a set of related constraints.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, QOSes are used to define different levels of access to the server&amp;#039;s resources. When [[User Jobs|executing a job with SLURM]], a user must always &amp;#039;&amp;#039;&amp;#039;specify the QOS&amp;#039;&amp;#039;&amp;#039; that their job will use: this choice, in turn, determines what resources the job is able to access and influences the [[#Job priority|priority]] of the job.&lt;br /&gt;
&lt;br /&gt;
Mufasa&amp;#039;s QOSes and their features can be inspected with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list qos format=name%-11,priority,MaxSubmit,maxwall,maxtres%-80,mintres%-18&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following (see below for the same data in table form):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
Name          Priority MaxSubmit     MaxWall MaxTRES                                                                          MinTRES            &lt;br /&gt;
----------- ---------- --------- ----------- -------------------------------------------------------------------------------- ------------------ &lt;br /&gt;
normal               0                                                                                                                           &lt;br /&gt;
nogpu                4         1  3-00:00:00 cpu=16,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=128G                    &lt;br /&gt;
gpuheavy-20          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=2,mem=128G             gres/gpu:4g.20gb=1 &lt;br /&gt;
gpuheavy-40          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=1,gres/gpu:4g.20gb=0,mem=128G             gres/gpu:40gb=1    &lt;br /&gt;
gpulight             8         1    12:00:00 cpu=2,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpu                  2         1  1-00:00:00 cpu=8,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpuwide              2         2  1-00:00:00 cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=1,mem=64G              gres/gpu:4g.20gb=1 &lt;br /&gt;
build               32         1    02:00:00 cpu=2,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=16G&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The columns of this output are the following:&lt;br /&gt;
&lt;br /&gt;
:; Name&lt;br /&gt;
:: name of the QOS&lt;br /&gt;
&lt;br /&gt;
:; Priority&lt;br /&gt;
:: priority tier associated to the QOS (higher value = higher priority): see [[#Job priority|Job priority]] for details&lt;br /&gt;
&lt;br /&gt;
:; MaxSubmit&lt;br /&gt;
:: maximum number of jobs from a single user that can be submitted to SLURM with this QOS; submitted jobs include both running and queued jobs&lt;br /&gt;
:: See [[#Limits on jobs by the same user|Limits on jobs by the same user]] for an overview of the limits on jobs set by Mufasa.&lt;br /&gt;
&lt;br /&gt;
:; MaxWall&lt;br /&gt;
:: maximum wall clock duration of the jobs using the QOS (after which they are killed by SLURM), in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
:: For some QOSes these are not set: it means that they are determined by the [[#SLURM partitions|partition]]. Partitions also define the [[#Default values|default duration]] of jobs.&lt;br /&gt;
&lt;br /&gt;
:; MaxTRES&lt;br /&gt;
:: amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) available to a job using the QOS, where&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;cpu=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of CPUs (i.e., processor cores) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of CPUs specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;gres/&amp;#039;&amp;#039;gpu:Type&amp;#039;&amp;#039;=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of GPUs of class &amp;lt;code&amp;gt;&amp;#039;&amp;#039;Type&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; (see [[User Jobs#gres syntax|&amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax]]) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; (for QOSes that allow access to GPUs) if not specified, the job cannot be launched&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;mem=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;G&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum amount of system RAM is &amp;#039;&amp;#039;K&amp;#039;&amp;#039; GBytes&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of RAM specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
&lt;br /&gt;
:; MinTRES&lt;br /&gt;
:: minimum amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) that a job using the QOS must request in order to actually get executed by SLURM.&lt;br /&gt;
:: (If your job does not actually need these resources, you&amp;#039;ve chosen the wrong QOS: you can use one with a higher priority.)&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; QOS is the default one, and exists only to ensure that users always specify a QOS when running a job. Since &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; has zero priority and no resources, a job run using this QOS would never be run.&lt;br /&gt;
&lt;br /&gt;
The information provided by the &amp;lt;code&amp;gt;sacctmgr list qos&amp;lt;/code&amp;gt; command above is summarised by the following table:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! QOS&lt;br /&gt;
! Priority&amp;lt;/br&amp;gt;tier&lt;br /&gt;
! Max&amp;lt;/br&amp;gt;Submit&lt;br /&gt;
! MaxWall&amp;lt;/br&amp;gt;[h]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;CPUs&lt;br /&gt;
! max RAM&amp;lt;/br&amp;gt;[GB]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;3g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;4g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;40GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | build&lt;br /&gt;
| 32&lt;br /&gt;
| 1&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 16&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpulight&lt;br /&gt;
| 8&lt;br /&gt;
| 1&lt;br /&gt;
| 12&lt;br /&gt;
| 2&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | nogpu&lt;br /&gt;
| 4&lt;br /&gt;
| 1&lt;br /&gt;
| 72&lt;br /&gt;
| 16&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpu&lt;br /&gt;
| 2&lt;br /&gt;
| 1&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuwide&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-20&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| 2 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-40&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
A key piece of information in the table above is the &amp;#039;&amp;#039;&amp;#039;priority tier&amp;#039;&amp;#039;&amp;#039; associated to each QOS. In Mufasa 2.0, priority tiers are used to encourage users to use the &amp;#039;&amp;#039;&amp;#039;least powerful QOS that is compatible with their needs&amp;#039;&amp;#039;&amp;#039;, where &amp;quot;powerful&amp;quot; means &amp;quot;rich with resources&amp;quot;. Less powerful QOSes increase the priority of the jobs that use them, so these jobs tend to be executed sooner. See [[#Job priority|Job priority]] for details about how priority affects the execution order of jobs in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
== The &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS ==&lt;br /&gt;
&lt;br /&gt;
This QOS is specifically designed to be used by Mufasa users to &amp;#039;&amp;#039;&amp;#039;quickly build [[System#Containers|container images]]&amp;#039;&amp;#039;&amp;#039;. Its associated priority tier is very high: this, combined with the fact that jobs using this QOS require few resources, means that such jobs usually get executed very soon.&lt;br /&gt;
&lt;br /&gt;
On the other hand, the limited resources and the lack of access to the GPUs make the &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS unsuitable for other tasks.&lt;br /&gt;
&lt;br /&gt;
See [[Singularity#Building Singularity images|Building Singularity images]] for directions about building Singularity container images.&lt;br /&gt;
&lt;br /&gt;
== Restricted QOSes ==&lt;br /&gt;
&lt;br /&gt;
In Mufasa, the most powerful QOSes are reserved to researchers (called &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users: these include academic personnel and Ph.D. students). M.Sc. students, called &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users, cannot use them while running jobs.&lt;br /&gt;
&lt;br /&gt;
See [[#research users and students users|below]] to understand how &amp;#039;&amp;#039;user categories&amp;#039;&amp;#039; work in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
= &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users =&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa belong to two &amp;#039;&amp;#039;&amp;#039;user categories&amp;#039;&amp;#039;&amp;#039;. Categories provide users with different access to  to Mufasa&amp;#039;s resources: the idea being to provide researchers with more access while still letting students benefit from the server&amp;#039;s capabilities.&lt;br /&gt;
&lt;br /&gt;
User categories in Mufasa are the following:&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. academic personnel and Ph.D. students&lt;br /&gt;
::: * can use all QOSes, including the the [[#Restricted QOSes|restricted ones]]&lt;br /&gt;
::: * their jobs have a higher &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a higher number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. M.Sc. students&lt;br /&gt;
::: * cannot use the [[#Restricted QOSes|restricted QOSes]]&lt;br /&gt;
::: * their jobs have a lower &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a lower number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
You can inspect the differences in priority and running jobs between &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association format=&amp;quot;account,priority,maxjobs&amp;quot; | grep -E &amp;#039;Account|research|students&amp;#039;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
   Account   Priority MaxJobs &lt;br /&gt;
  research          4       2 &lt;br /&gt;
  students          1       1&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
To know what limits apply to your own user, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association where user=$USER format=&amp;quot;user,priority,maxjobs,qos%-60&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
      User   Priority MaxJobs QOS                                                          &lt;br /&gt;
---------- ---------- ------- ------------------------------------------------------------ &lt;br /&gt;
    preali          4       2 build,gpu,gpuheavy-20,gpuheavy-40,gpulight,gpuwide,nogpu&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The list under &amp;quot;QOS&amp;quot; shows what QOSes your user is allowed to use when [[User Jobs|running jobs]]. &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users can use all of them, while &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users can only access a subset of them.&lt;br /&gt;
&lt;br /&gt;
= Limits on jobs by the same user =&lt;br /&gt;
&lt;br /&gt;
Mufasa enforces limits on the number of jobs from a single user. Such limits aim at preventing users from &amp;quot;hogging&amp;quot; system resources, and apply to:&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;submitted jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that the user asked SLURM to execute, each of which may currently be either running or queued&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;running jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that are currently in execution&lt;br /&gt;
&lt;br /&gt;
Limits on submitted jobs are set via [[#SLURM Quality of Service (QOS)|QOSes]], while limits on running jobs are set via [[#research users and students users|user category]].&lt;br /&gt;
&lt;br /&gt;
The following table summarises what limits exist:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! &lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;running&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;submitted&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | global limits&amp;lt;br/&amp;gt;(system-wide)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to limits set by individual QOSes (below)&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | limits for&amp;lt;br/&amp;gt;individual&amp;lt;/br&amp;gt;QOSes&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to global limits (above)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for the&amp;#039;&amp;#039; &amp;lt;code&amp;gt;gpuwide&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;QOS&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for each of the other QOSes&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
= Job priority =&lt;br /&gt;
&lt;br /&gt;
Once the execution of a job has been requested, the job is not run immediately: it is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039; by SLURM, together with all the other jobs awaiting execution. The job on top of the queue at any time is the first to be put into execution as soon as the resources it requires are available. &lt;br /&gt;
&lt;br /&gt;
The order of the items in the job queue depends on their &amp;#039;&amp;#039;&amp;#039;priority&amp;#039;&amp;#039;&amp;#039;. The job with the highest priority is put by SLURM on top of the queue, while all other jobs are put in the queue in order of descending priority.&lt;br /&gt;
&lt;br /&gt;
The goal of SLURM is to maximise resource availability: i.e., to ensure the shorter possible wait time before job execution. To achieve this goal, in Mufasa SLURM is configured to &amp;#039;&amp;#039;&amp;#039;encourage users not to ask for resources or execution time that their job doesn&amp;#039;t need&amp;#039;&amp;#039;&amp;#039;. This is done via the priority mechanism: the more resources and/or time a job requests, the lower its priority will be; and the later it will be executed.&lt;br /&gt;
&lt;br /&gt;
Priority management in Mufasa is designed to set up a &amp;#039;&amp;#039;&amp;#039;virtuous cycle&amp;#039;&amp;#039;&amp;#039; where users, by carefully choosing what to ask for, obtain two results:&lt;br /&gt;
* they ensure that their job is executed as soon as possible;&lt;br /&gt;
* they leave as much as possible of Mufasa&amp;#039;s resources free for other users&amp;#039;s jobs.&lt;br /&gt;
&lt;br /&gt;
== Elements determining job priority ==&lt;br /&gt;
In Mufasa, the priority of a job is computed by SLURM according to the following elements:&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#research users and students users|User category]]&amp;#039;&amp;#039;&amp;#039; (i.e., &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;)&lt;br /&gt;
::: Used to provide higher priority to jobs run by &amp;#039;&amp;#039;&amp;#039;research personnel&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#SLURM Quality of Service (QOS)|QOS]]&amp;#039;&amp;#039;&amp;#039; used by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;less resources&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Number of CPUs&amp;#039;&amp;#039;&amp;#039; requested by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;a lower number of CPUs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job Age&amp;#039;&amp;#039;&amp;#039;, i.e. the time that the job has been waiting in the queue&lt;br /&gt;
::: Used to provide higher priority to jobs which have been &amp;#039;&amp;#039;&amp;#039;queued for a longer time&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;FairShare&amp;#039;&amp;#039;&amp;#039;, i.e. a factor computed by SLURM to balance use of the system by different users&lt;br /&gt;
::: Used to provide higher priority to jobs by users who &amp;#039;&amp;#039;&amp;#039;used Mufasa&amp;#039;s resources less than others&amp;#039;&amp;#039;&amp;#039; (see [[#How FairShare works|below]])&lt;br /&gt;
&lt;br /&gt;
You can understand how each of these elements contributes to the priority of the jobs waiting for execution (both yours and of other users) with command &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;. A tailored version of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; that only displays the parameters listed above is the following:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
prio -o &amp;quot;%6i %9u %.8Y %.10B %.5Q %.9T %.5A %.5F&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Here is an example of the output of the command above:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
JOBID  USER      PRIORITY      ASSOC   QOS      TRES   AGE FAIRS&lt;br /&gt;
5774   itombini      3576       2500   750  cpu=-800   990   136&lt;br /&gt;
5780   fbottazzi     2350        625   750  cpu=-600   504  1071&lt;br /&gt;
5783   mzoccola      1849        625   750  cpu=-800   300   974&lt;br /&gt;
5790   amoglia       2801       2500   375  cpu=-800   239   487&lt;br /&gt;
5791   qliu          2315       2500   375  cpu=-800   221    19&lt;br /&gt;
5795   lpassoni      2600       2500   750  cpu=-800    92    58&lt;br /&gt;
5805   adurat        3154       2500   750  cpu=-800    62   643&lt;br /&gt;
5806   itombini      2645       2500   750  cpu=-800    59   136&lt;br /&gt;
5813   fpastori      5385       2500  3000  cpu=-200    47    39&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
In it, column PRIORITY provides the overall priority of each job. Each priority value is the sum of the five values to its right; in particular, column ASSOC corresponds to user category, while column TRES corresponds to the number of CPUs. Note that the contribution of TRES to priority is &amp;#039;&amp;#039;negative&amp;#039;&amp;#039;, i.e. it is a penalty; its value is proportional to the number of CPUs requested by the job.&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;Things to consider when using &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
* Its output is only relevant for jobs &amp;#039;&amp;#039;not yet in execution&amp;#039;&amp;#039;. For an already running job, the values shown by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; correspond to the priority that the job would get if evaluated now, which generally differs from its actual priority.&lt;br /&gt;
* Jobs listed by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; may temporarily differ from jobs listed by &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;: this is due to the fact that SLURM updates its internal data with a periodicity of a few minutes.&lt;br /&gt;
&lt;br /&gt;
=== How FairShare works ===&lt;br /&gt;
In Mufasa 2.0, the FairShare of each user is computed according to these rules:&lt;br /&gt;
# It considers the quantity used by (current and past) user jobs of:&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;CPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of CPUs used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;RAM&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the amount of RAM used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;GPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of GPUs used&lt;br /&gt;
#** note: 40 GB GPUs have double the impact on FairShare than 20 GB GPUs&lt;br /&gt;
# The impact on FairShare of any use of resources is &amp;#039;&amp;#039;&amp;#039;proportional to its duration&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
#* example: using 32 GB of RAM for 48h has the same impact of using 64 GB of RAM for 24h&lt;br /&gt;
# FairShare has a &amp;quot;fading memory&amp;quot;: i.e., resource use has less and less impact on FairShare the farther in the past it is&lt;br /&gt;
#* consequence: over time, the &amp;quot;history&amp;quot; of a user gets forgotten by FairShare&lt;br /&gt;
&lt;br /&gt;
== How to maximise the priority of your jobs ==&lt;br /&gt;
&lt;br /&gt;
Every time you run a SLURM job, follow these guidelines:&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Choose the less powerful QOS compatible with the needs of your job&lt;br /&gt;
:: QOSes with access to less resources have higher base priority&lt;br /&gt;
&lt;br /&gt;
; Only request CPUs and RAM that your job really needs&lt;br /&gt;
:: Usually, code exploits multiple CPUs only if specifically designed to do so: if unsure, only ask for 1 CPU&lt;br /&gt;
:: The fewer resources your job asks for, the less it will wait before they become available&lt;br /&gt;
:: The more CPUs your job asks for, the stronger the penalty to its priority&lt;br /&gt;
:: Asking for fewer CPUs and/or less RAM improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Do not request more time than your jobs needs to complete&lt;br /&gt;
:: Make a worst-case estimate and only ask for that duration&lt;br /&gt;
:: Asking for less of Mufasa&amp;#039;s time improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Debug and test code using less powerful QOSes before running it with more powerful QOSes&lt;br /&gt;
:: Your test jobs will get a higher priority and your FairShare will improve&lt;br /&gt;
&lt;br /&gt;
; Cancel jobs that you don&amp;#039;t need them anymore&lt;br /&gt;
:: Always use [[User_Jobs#Cancelling_a_job_with_scancel|scancel]] to delete completed (or crashed) jobs: your Fairshare will improve&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Protip: it&amp;#039;s a good idea to [[User Jobs#Looking for unused GPUs|check for unused GPUs]] before choosing what to request. Requesting a GPU that is currently idle will help your job get executed sooner.&lt;br /&gt;
&lt;br /&gt;
= SLURM partitions =&lt;br /&gt;
&lt;br /&gt;
Partitions are another mechanism provided by SLURM to create different levels of access to system resources. Since in Mufasa 2.0 access to resources is controlled via [[#SLURM Quality of Service (QOS)|QOSes]], partitions are not very relevant. (This is one of the main differences between Mufasa 1.0 and Mufasa 2.0.)&lt;br /&gt;
&lt;br /&gt;
Note, however, that the default values for some features of SLURM jobs (e.g., duration) are [[#Default values|set by the partition]].&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, there is a single SLURM partition, called &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt;, and all jobs run on it. The state of &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt; can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sinfo -o &amp;quot;%10P %5a %9T %11L %10l&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PARTITION  AVAIL STATE     DEFAULTTIME TIMELIMIT &lt;br /&gt;
jobs*      up    idle      1:00:00     3-00:00:00&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where columns correspond to the following information:&lt;br /&gt;
&lt;br /&gt;
:; PARTITION&lt;br /&gt;
:: name of the partition; the asterisks indicates that it&amp;#039;s the default one&lt;br /&gt;
&lt;br /&gt;
:; AVAIL&lt;br /&gt;
:: state/availability of the partition: see [[#Partition availability|below]]&lt;br /&gt;
&lt;br /&gt;
:; STATE&lt;br /&gt;
:: state (using [https://slurm.schedmd.com/sinfo.html#SECTION_NODE-STATE-CODES these codes])&lt;br /&gt;
:: typical values are &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;mixed&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that some of the resources are busy executing jobs while other are idle, and &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;allocated&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that all of the resources are in use&lt;br /&gt;
&lt;br /&gt;
:; DEFAULTTIME&lt;br /&gt;
:: default runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:; TIMELIMIT&lt;br /&gt;
:: maximum runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
The asterisk at the end of the partition name indicates the default partition, i.e. the one on which jobs which do not ask for a specific partition are run.&lt;br /&gt;
&lt;br /&gt;
Command &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; does not tell you about the &amp;#039;&amp;#039;jobs&amp;#039;&amp;#039; submitted to a partition. This information is obtained, instead, with [[User Jobs#Inspecting jobs with squeue|command &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;]].&lt;br /&gt;
&lt;br /&gt;
== Partition availability ==&lt;br /&gt;
&lt;br /&gt;
The most important information that &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; provides is the &amp;#039;&amp;#039;&amp;#039;availability&amp;#039;&amp;#039;&amp;#039; (also called &amp;#039;&amp;#039;&amp;#039;state&amp;#039;&amp;#039;&amp;#039;) of partitions. This is shown in column &amp;quot;AVAIL&amp;quot;. Possible partition states are:&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is available&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed as soon as resources allow&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is in the process of becoming unavailable (i.e., of entering the &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; state: see below)&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is unavailable&lt;br /&gt;
:: There are no running jobs&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
When a partition goes from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; no harm is done to running jobs. In a normally functioning SLURM system, the passage from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; happens only when no jobs are running on the partition. If (e.g., due to a malfunction) the passage happens with jobs still running, they get killed.&lt;br /&gt;
&lt;br /&gt;
A partition in state &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; requires intervention by a [[Roles|Job Administrator]] to be restored to &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
== Default values ==&lt;br /&gt;
&lt;br /&gt;
The features of SLURM partitions, including the &amp;#039;&amp;#039;&amp;#039;default values&amp;#039;&amp;#039;&amp;#039; which are applied to jobs that do not make explicit requests, can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scontrol show partition&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PartitionName=jobs&lt;br /&gt;
   AllowGroups=ALL AllowAccounts=ALL AllowQos=nogpu,gpulight,gpu,gpuwide,gpuheavy-20,gpuheavy-40&lt;br /&gt;
   AllocNodes=ALL Default=YES QoS=N/A&lt;br /&gt;
   DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO&lt;br /&gt;
   MaxNodes=UNLIMITED MaxTime=3-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED&lt;br /&gt;
   Nodes=gn01&lt;br /&gt;
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO&lt;br /&gt;
   OverTimeLimit=NONE PreemptMode=OFF&lt;br /&gt;
   State=UP TotalCPUs=48 TotalNodes=1 SelectTypeParameters=NONE&lt;br /&gt;
   JobDefaults=(null)&lt;br /&gt;
   DefMemPerNode=4096 MaxMemPerNode=UNLIMITED&lt;br /&gt;
   TRES=cpu=48,mem=1011435M,node=1,billing=49,gres/gpu=13,gres/gpu:3g.20gb=5,gres/gpu:40gb=3,gres/gpu:4g.20gb=5&lt;br /&gt;
   TRESBillingWeights=cpu=1.0,gres/gpu:3g.20gb=6.0,gres/gpu:4g.20gb=6.0,gres/gpu:40gb=6.0,mem=0.05g&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In the output of &amp;lt;code&amp;gt;scontrol show partition&amp;lt;/code&amp;gt; the most relevant default values for Mufasa users are:&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefaultTime&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default execution time assigned to a job run on the partition (e.g., 1 hour)&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefMemPerNode&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default amount of RAM assigned to a job run on the partition (e.g., 4GB)&lt;br /&gt;
&lt;br /&gt;
= System resources subjected to limitations =&lt;br /&gt;
&lt;br /&gt;
In systems based on SLURM like Mufasa, &amp;#039;&amp;#039;&amp;#039;TRES (Trackable RESources)&amp;#039;&amp;#039;&amp;#039; are (from [https://slurm.schedmd.com/tres.html SLURM&amp;#039;s documentation] &amp;quot;&amp;#039;&amp;#039;resources that can be tracked for usage or used to enforce limits against.&amp;#039;&amp;#039;&amp;quot;&lt;br /&gt;
&lt;br /&gt;
TRES include CPUs, RAM and &amp;#039;&amp;#039;&amp;#039;GRES&amp;#039;&amp;#039;&amp;#039;. The last term stands for &amp;#039;&amp;#039;Generic RESources&amp;#039;&amp;#039; that a job may need for its execution. In Mufasa, the only &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; resources are the GPUs.&lt;br /&gt;
&lt;br /&gt;
== &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax ==&lt;br /&gt;
&lt;br /&gt;
To ask SLURM to assign GRES resources (i.e., GPUs) to a job, a special syntax must be used. Precisely, the name of each GPU resource takes the form&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:Name:Type&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
Considering the [[System#CPUs and GPUs|GPU complement of Mufasa]], &amp;lt;code&amp;gt;Type&amp;lt;/code&amp;gt; takes the following values:&lt;br /&gt;
&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:40gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 40 Gbytes of RAM&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:4g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 4 compute units&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 3 compute units&lt;br /&gt;
&lt;br /&gt;
So, for instance,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
identifies a resource corresponding to a GPU with 20 GB of RAM and 3 compute units.&lt;br /&gt;
&lt;br /&gt;
When asking for a GRES resource (e.g., in an &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command or an &amp;lt;code&amp;gt;SBATCH&amp;lt;/code&amp;gt; directive of an [[User Jobs#Using execution scripts to run jobs|execution script]]), the syntax required by SLURM is&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:&amp;lt;Type&amp;gt;:&amp;lt;Quantity&amp;gt;&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
where &amp;lt;code&amp;gt;Quantity&amp;lt;/code&amp;gt; is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of type &amp;lt;code&amp;gt;4g.20gb&amp;lt;/code&amp;gt; the syntax is&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:4g.20gb:2&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SLURM&amp;#039;s &amp;#039;&amp;#039;generic resources&amp;#039;&amp;#039; are defined in &amp;lt;code&amp;gt;/etc/slurm/gres.conf&amp;lt;/code&amp;gt;. In order to make GPUs available to SLURM&amp;#039;s &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; management, Mufasa makes use of Nvidia&amp;#039;s [https://developer.nvidia.com/nvidia-management-library-nvml NVML library]. For additional information see [https://slurm.schedmd.com/gres.html SLURM&amp;#039;s documentation].&lt;/div&gt;</summary>
		<author><name>GiulioFontana</name></author>
	</entry>
	<entry>
		<id>https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2589</id>
		<title>SLURM</title>
		<link rel="alternate" type="text/html" href="https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2589"/>
		<updated>2026-11-04T09:04:46Z</updated>

		<summary type="html">&lt;p&gt;GiulioFontana: /* Elements determining job priority */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page presents the features of SLURM that are most relevant to Mufasa&amp;#039;s [[Roles|Job Users]]. Job Users can submit jobs for execution, cancel their own jobs, and see other users&amp;#039; jobs (but not intervene on them).&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa &amp;#039;&amp;#039;&amp;#039;must use SLURM&amp;#039;&amp;#039;&amp;#039; to run resource-heavy processes, i.e. computing jobs that require one or more of the following:&lt;br /&gt;
* GPUs&lt;br /&gt;
* multiple CPUs&lt;br /&gt;
* powerful CPUs&lt;br /&gt;
* a significant amount of RAM&lt;br /&gt;
&lt;br /&gt;
In fact, only processes run via SLURM have access to all the resources of Mufasa. Processes run outside SLURM are executed by the [[System#Login server|login server]] virtual machine, which has minimal resources and no GPUs. Using SLURM is therefore the only way to execute resource-heavy jobs on Mufasa (this is a key difference between Mufasa 1.0 and Mufasa 2.0).&lt;br /&gt;
&lt;br /&gt;
= SLURM in a nutshell =&lt;br /&gt;
&lt;br /&gt;
Computation jobs on Mufasa needs to be launched via [[System#The SLURM job scheduling system|SLURM]]. SLURM provides jobs with access to the [[#System resources subjected to limitations|physical resources]] of Mufasa, such as CPUs, GPUs and RAM. Thanks to SLURM, processing jobs share system resources, optimising their occupation and availability. &lt;br /&gt;
&lt;br /&gt;
When a user runs a job, the job does not get executed immediately and is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039;. SLURM executes jobs according to their order in the queue: the top job in the queue gets executed as soon as the necessary resources are available, while jobs lower in the queue wait longer. The position of a job in the queue is due to the &amp;#039;&amp;#039;&amp;#039;[[#Job priority|priority]]&amp;#039;&amp;#039;&amp;#039; assigned to it by SLURM, with higher-priority jobs closer to the top. As a general rule:&lt;br /&gt;
&lt;br /&gt;
;: &amp;#039;&amp;#039;&amp;#039;the greater the fraction of Mufasa&amp;#039;s overall resources that a job asks for, the lower the job&amp;#039;s priority will be&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
The priority mechanism is used to encourage users to use Mufasa&amp;#039;s resources (i.e.: GPUs, CPUs, RAM, execution time) in an effective and equitable manner. This page includes a [[#How_to_maximise_the_priority_of_your_jobs|chart explaining how to maximise the priority of your jobs]].&lt;br /&gt;
&lt;br /&gt;
The &amp;#039;&amp;#039;&amp;#039;time&amp;#039;&amp;#039;&amp;#039; available to a job for its execution is controlled by SLURM. When a user requests execution of a job, they must specify the duration of the time slot that the job needs. The job must complete its execution before the end of the requested time slot, because at the end of such slot it gets killed by SLURM.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0 access to system resources is managed via SLURM&amp;#039;s [[#SLURM Quality of Service (QOS)|Quality of Service (QOS)]] mechanism (Mufasa 1.0 used [[#SLURM_partitions|partitions]] instead). While [[User Jobs#Running jobs with SLURM|launching a processing job via SLURM]], the user must always specify the chosen QOS. QOSes differ in the set of resources that they provide access to because each of them is designed to fit a given type of job.&lt;br /&gt;
&lt;br /&gt;
Mufasa sets limits to the number of jobs by the same user. This page includes a [[#Limits on jobs by the same user|table summarising such limits]].&lt;br /&gt;
&lt;br /&gt;
= SLURM Quality of Service (QOS) =&lt;br /&gt;
&lt;br /&gt;
Through &amp;#039;&amp;#039;&amp;#039;Quality of Services&amp;#039;&amp;#039;&amp;#039; (&amp;#039;&amp;#039;&amp;#039;QOSes&amp;#039;&amp;#039;&amp;#039;), SLURM lets system configurators assign a name to a set of related constraints.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, QOSes are used to define different levels of access to the server&amp;#039;s resources. When [[User Jobs|executing a job with SLURM]], a user must always &amp;#039;&amp;#039;&amp;#039;specify the QOS&amp;#039;&amp;#039;&amp;#039; that their job will use: this choice, in turn, determines what resources the job is able to access and influences the [[#Job priority|priority]] of the job.&lt;br /&gt;
&lt;br /&gt;
Mufasa&amp;#039;s QOSes and their features can be inspected with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list qos format=name%-11,priority,MaxSubmit,maxwall,maxtres%-80,mintres%-18&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following (see below for the same data in table form):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
Name          Priority MaxSubmit     MaxWall MaxTRES                                                                          MinTRES            &lt;br /&gt;
----------- ---------- --------- ----------- -------------------------------------------------------------------------------- ------------------ &lt;br /&gt;
normal               0                                                                                                                           &lt;br /&gt;
nogpu                4         1  3-00:00:00 cpu=16,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=128G                    &lt;br /&gt;
gpuheavy-20          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=2,mem=128G             gres/gpu:4g.20gb=1 &lt;br /&gt;
gpuheavy-40          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=1,gres/gpu:4g.20gb=0,mem=128G             gres/gpu:40gb=1    &lt;br /&gt;
gpulight             8         1    12:00:00 cpu=2,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpu                  2         1  1-00:00:00 cpu=8,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpuwide              2         2  1-00:00:00 cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=1,mem=64G              gres/gpu:4g.20gb=1 &lt;br /&gt;
build               32         1    02:00:00 cpu=2,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=16G&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The columns of this output are the following:&lt;br /&gt;
&lt;br /&gt;
:; Name&lt;br /&gt;
:: name of the QOS&lt;br /&gt;
&lt;br /&gt;
:; Priority&lt;br /&gt;
:: priority tier associated to the QOS (higher value = higher priority): see [[#Job priority|Job priority]] for details&lt;br /&gt;
&lt;br /&gt;
:; MaxSubmit&lt;br /&gt;
:: maximum number of jobs from a single user that can be submitted to SLURM with this QOS; submitted jobs include both running and queued jobs&lt;br /&gt;
:: See [[#Limits on jobs by the same user|Limits on jobs by the same user]] for an overview of the limits on jobs set by Mufasa.&lt;br /&gt;
&lt;br /&gt;
:; MaxWall&lt;br /&gt;
:: maximum wall clock duration of the jobs using the QOS (after which they are killed by SLURM), in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
:: For some QOSes these are not set: it means that they are determined by the [[#SLURM partitions|partition]]. Partitions also define the [[#Default values|default duration]] of jobs.&lt;br /&gt;
&lt;br /&gt;
:; MaxTRES&lt;br /&gt;
:: amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) available to a job using the QOS, where&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;cpu=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of CPUs (i.e., processor cores) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of CPUs specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;gres/&amp;#039;&amp;#039;gpu:Type&amp;#039;&amp;#039;=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of GPUs of class &amp;lt;code&amp;gt;&amp;#039;&amp;#039;Type&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; (see [[User Jobs#gres syntax|&amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax]]) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; (for QOSes that allow access to GPUs) if not specified, the job cannot be launched&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;mem=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;G&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum amount of system RAM is &amp;#039;&amp;#039;K&amp;#039;&amp;#039; GBytes&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of RAM specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
&lt;br /&gt;
:; MinTRES&lt;br /&gt;
:: minimum amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) that a job using the QOS must request in order to actually get executed by SLURM.&lt;br /&gt;
:: (If your job does not actually need these resources, you&amp;#039;ve chosen the wrong QOS: you can use one with a higher priority.)&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; QOS is the default one, and exists only to ensure that users always specify a QOS when running a job. Since &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; has zero priority and no resources, a job run using this QOS would never be run.&lt;br /&gt;
&lt;br /&gt;
The information provided by the &amp;lt;code&amp;gt;sacctmgr list qos&amp;lt;/code&amp;gt; command above is summarised by the following table:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! QOS&lt;br /&gt;
! Priority&amp;lt;/br&amp;gt;tier&lt;br /&gt;
! Max&amp;lt;/br&amp;gt;Submit&lt;br /&gt;
! MaxWall&amp;lt;/br&amp;gt;[h]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;CPUs&lt;br /&gt;
! max RAM&amp;lt;/br&amp;gt;[GB]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;3g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;4g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;40GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | build&lt;br /&gt;
| 32&lt;br /&gt;
| 1&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 16&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpulight&lt;br /&gt;
| 8&lt;br /&gt;
| 1&lt;br /&gt;
| 12&lt;br /&gt;
| 2&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | nogpu&lt;br /&gt;
| 4&lt;br /&gt;
| 1&lt;br /&gt;
| 72&lt;br /&gt;
| 16&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpu&lt;br /&gt;
| 2&lt;br /&gt;
| 1&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuwide&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-20&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| 2 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-40&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
A key piece of information in the table above is the &amp;#039;&amp;#039;&amp;#039;priority tier&amp;#039;&amp;#039;&amp;#039; associated to each QOS. In Mufasa 2.0, priority tiers are used to encourage users to use the &amp;#039;&amp;#039;&amp;#039;least powerful QOS that is compatible with their needs&amp;#039;&amp;#039;&amp;#039;, where &amp;quot;powerful&amp;quot; means &amp;quot;rich with resources&amp;quot;. Less powerful QOSes increase the priority of the jobs that use them, so these jobs tend to be executed sooner. See [[#Job priority|Job priority]] for details about how priority affects the execution order of jobs in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
== The &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS ==&lt;br /&gt;
&lt;br /&gt;
This QOS is specifically designed to be used by Mufasa users to &amp;#039;&amp;#039;&amp;#039;quickly build [[System#Containers|container images]]&amp;#039;&amp;#039;&amp;#039;. Its associated priority tier is very high: this, combined with the fact that jobs using this QOS require few resources, means that such jobs usually get executed very soon.&lt;br /&gt;
&lt;br /&gt;
On the other hand, the limited resources and the lack of access to the GPUs make the &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS unsuitable for other tasks.&lt;br /&gt;
&lt;br /&gt;
See [[Singularity#Building Singularity images|Building Singularity images]] for directions about building Singularity container images.&lt;br /&gt;
&lt;br /&gt;
== Restricted QOSes ==&lt;br /&gt;
&lt;br /&gt;
In Mufasa, the most powerful QOSes are reserved to researchers (called &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users: these include academic personnel and Ph.D. students). M.Sc. students, called &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users, cannot use them while running jobs.&lt;br /&gt;
&lt;br /&gt;
See [[#research users and students users|below]] to understand how &amp;#039;&amp;#039;user categories&amp;#039;&amp;#039; work in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
= &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users =&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa belong to two &amp;#039;&amp;#039;&amp;#039;user categories&amp;#039;&amp;#039;&amp;#039;. Categories provide users with different access to  to Mufasa&amp;#039;s resources: the idea being to provide researchers with more access while still letting students benefit from the server&amp;#039;s capabilities.&lt;br /&gt;
&lt;br /&gt;
User categories in Mufasa are the following:&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. academic personnel and Ph.D. students&lt;br /&gt;
::: * can use all QOSes, including the the [[#Restricted QOSes|restricted ones]]&lt;br /&gt;
::: * their jobs have a higher &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a higher number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. M.Sc. students&lt;br /&gt;
::: * cannot use the [[#Restricted QOSes|restricted QOSes]]&lt;br /&gt;
::: * their jobs have a lower &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a lower number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
You can inspect the differences in priority and running jobs between &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association format=&amp;quot;account,priority,maxjobs&amp;quot; | grep -E &amp;#039;Account|research|students&amp;#039;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
   Account   Priority MaxJobs &lt;br /&gt;
  research          4       2 &lt;br /&gt;
  students          1       1&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
To know what limits apply to your own user, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association where user=$USER format=&amp;quot;user,priority,maxjobs,qos%-60&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
      User   Priority MaxJobs QOS                                                          &lt;br /&gt;
---------- ---------- ------- ------------------------------------------------------------ &lt;br /&gt;
    preali          4       2 build,gpu,gpuheavy-20,gpuheavy-40,gpulight,gpuwide,nogpu&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The list under &amp;quot;QOS&amp;quot; shows what QOSes your user is allowed to use when [[User Jobs|running jobs]]. &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users can use all of them, while &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users can only access a subset of them.&lt;br /&gt;
&lt;br /&gt;
= Limits on jobs by the same user =&lt;br /&gt;
&lt;br /&gt;
Mufasa enforces limits on the number of jobs from a single user. Such limits aim at preventing users from &amp;quot;hogging&amp;quot; system resources, and apply to:&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;submitted jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that the user asked SLURM to execute, each of which may currently be either running or queued&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;running jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that are currently in execution&lt;br /&gt;
&lt;br /&gt;
Limits on submitted jobs are set via [[#SLURM Quality of Service (QOS)|QOSes]], while limits on running jobs are set via [[#research users and students users|user category]].&lt;br /&gt;
&lt;br /&gt;
The following table summarises what limits exist:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! &lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;running&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;submitted&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | global limits&amp;lt;br/&amp;gt;(system-wide)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to limits set by individual QOSes (below)&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | limits for&amp;lt;br/&amp;gt;individual&amp;lt;/br&amp;gt;QOSes&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to global limits (above)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for the&amp;#039;&amp;#039; &amp;lt;code&amp;gt;gpuwide&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;QOS&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for each of the other QOSes&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
= Job priority =&lt;br /&gt;
&lt;br /&gt;
Once the execution of a job has been requested, the job is not run immediately: it is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039; by SLURM, together with all the other jobs awaiting execution. The job on top of the queue at any time is the first to be put into execution as soon as the resources it requires are available. &lt;br /&gt;
&lt;br /&gt;
The order of the items in the job queue depends on their &amp;#039;&amp;#039;&amp;#039;priority&amp;#039;&amp;#039;&amp;#039;. The job with the highest priority is put by SLURM on top of the queue, while all other jobs are put in the queue in order of descending priority.&lt;br /&gt;
&lt;br /&gt;
The goal of SLURM is to maximise resource availability: i.e., to ensure the shorter possible wait time before job execution. To achieve this goal, in Mufasa SLURM is configured to &amp;#039;&amp;#039;&amp;#039;encourage users not to ask for resources or execution time that their job doesn&amp;#039;t need&amp;#039;&amp;#039;&amp;#039;. This is done via the priority mechanism: the more resources and/or time a job requests, the lower its priority will be; and the later it will be executed.&lt;br /&gt;
&lt;br /&gt;
Priority management in Mufasa is designed to set up a &amp;#039;&amp;#039;&amp;#039;virtuous cycle&amp;#039;&amp;#039;&amp;#039; where users, by carefully choosing what to ask for, obtain two results:&lt;br /&gt;
* they ensure that their job is executed as soon as possible;&lt;br /&gt;
* they leave as much as possible of Mufasa&amp;#039;s resources free for other users&amp;#039;s jobs.&lt;br /&gt;
&lt;br /&gt;
== Elements determining job priority ==&lt;br /&gt;
In Mufasa, the priority of a job is computed by SLURM according to the following elements:&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#research users and students users|User category]]&amp;#039;&amp;#039;&amp;#039; (i.e., &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;)&lt;br /&gt;
::: Used to provide higher priority to jobs run by &amp;#039;&amp;#039;&amp;#039;research personnel&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#SLURM Quality of Service (QOS)|QOS]]&amp;#039;&amp;#039;&amp;#039; used by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;less resources&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Number of CPUs&amp;#039;&amp;#039;&amp;#039; requested by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;a lower number of CPUs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job Age&amp;#039;&amp;#039;&amp;#039;, i.e. the time that the job has been waiting in the queue&lt;br /&gt;
::: Used to provide higher priority to jobs which have been &amp;#039;&amp;#039;&amp;#039;queued for a longer time&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;FairShare&amp;#039;&amp;#039;&amp;#039;, i.e. a factor computed by SLURM to balance use of the system by different users&lt;br /&gt;
::: Used to provide higher priority to jobs by users who &amp;#039;&amp;#039;&amp;#039;used Mufasa&amp;#039;s resources less than others&amp;#039;&amp;#039;&amp;#039; (see [[#How FairShare works|below]])&lt;br /&gt;
&lt;br /&gt;
You can understand how each of these elements contributes to the priority of the jobs waiting for execution (both yours and of other users) with command &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;. A tailored version of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; that only displays the parameters listed above is the following:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
prio -o &amp;quot;%6i %9u %.8Y %.10B %.5Q %.9T %.5A %.5F&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Here is an example of the output of the command above:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
JOBID  USER      PRIORITY      ASSOC   QOS      TRES   AGE FAIRS&lt;br /&gt;
5774   itombini      3576       2500   750  cpu=-800   990   136&lt;br /&gt;
5780   fbottazzi     2350        625   750  cpu=-600   504  1071&lt;br /&gt;
5783   mzoccola      1849        625   750  cpu=-800   300   974&lt;br /&gt;
5790   amoglia       2801       2500   375  cpu=-800   239   487&lt;br /&gt;
5791   qliu          2315       2500   375  cpu=-800   221    19&lt;br /&gt;
5795   lpassoni      2600       2500   750  cpu=-800    92    58&lt;br /&gt;
5805   adurat        3154       2500   750  cpu=-800    62   643&lt;br /&gt;
5806   itombini      2645       2500   750  cpu=-800    59   136&lt;br /&gt;
5813   fpastori      5385       2500  3000  cpu=-200    47    39&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
In it, column PRIORITY provides the overall priority of each job. Each priority value is the sum of the five values to its right; in particular, column ASSOC corresponds to user category, while column TRES corresponds to the number of CPUs. Note that the latter contribution to priority is &amp;#039;&amp;#039;negative&amp;#039;&amp;#039;, i.e. it is a penalty.&lt;br /&gt;
&lt;br /&gt;
; Things to consider when using &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;&lt;br /&gt;
:: Its output is only relevant for jobs &amp;#039;&amp;#039;&amp;#039;not yet in execution&amp;#039;&amp;#039;&amp;#039;. For an already running job, the values shown by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; correspond to the priority that the job would get if evaluated now, which generally differs from its actual priority.&lt;br /&gt;
:: Jobs listed by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; may temporarily differ from jobs listed by &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;: this is due to the fact that SLURM updates its internal data with a periodicity of a few minutes.&lt;br /&gt;
&lt;br /&gt;
=== How FairShare works ===&lt;br /&gt;
In Mufasa 2.0, the FairShare of each user is computed according to these rules:&lt;br /&gt;
# It considers the quantity used by (current and past) user jobs of:&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;CPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of CPUs used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;RAM&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the amount of RAM used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;GPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of GPUs used&lt;br /&gt;
#** note: 40 GB GPUs have double the impact on FairShare than 20 GB GPUs&lt;br /&gt;
# The impact on FairShare of any use of resources is &amp;#039;&amp;#039;&amp;#039;proportional to its duration&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
#* example: using 32 GB of RAM for 48h has the same impact of using 64 GB of RAM for 24h&lt;br /&gt;
# FairShare has a &amp;quot;fading memory&amp;quot;: i.e., resource use has less and less impact on FairShare the farther in the past it is&lt;br /&gt;
#* consequence: over time, the &amp;quot;history&amp;quot; of a user gets forgotten by FairShare&lt;br /&gt;
&lt;br /&gt;
== How to maximise the priority of your jobs ==&lt;br /&gt;
&lt;br /&gt;
Every time you run a SLURM job, follow these guidelines:&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Choose the less powerful QOS compatible with the needs of your job&lt;br /&gt;
:: QOSes with access to less resources have higher base priority&lt;br /&gt;
&lt;br /&gt;
; Only request CPUs and RAM that your job really needs&lt;br /&gt;
:: Usually, code exploits multiple CPUs only if specifically designed to do so: if unsure, only ask for 1 CPU&lt;br /&gt;
:: The fewer resources your job asks for, the less it will wait before they become available&lt;br /&gt;
:: The more CPUs your job asks for, the stronger the penalty to its priority&lt;br /&gt;
:: Asking for fewer CPUs and/or less RAM improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Do not request more time than your jobs needs to complete&lt;br /&gt;
:: Make a worst-case estimate and only ask for that duration&lt;br /&gt;
:: Asking for less of Mufasa&amp;#039;s time improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Debug and test code using less powerful QOSes before running it with more powerful QOSes&lt;br /&gt;
:: Your test jobs will get a higher priority and your FairShare will improve&lt;br /&gt;
&lt;br /&gt;
; Cancel jobs that you don&amp;#039;t need them anymore&lt;br /&gt;
:: Always use [[User_Jobs#Cancelling_a_job_with_scancel|scancel]] to delete completed (or crashed) jobs: your Fairshare will improve&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Protip: it&amp;#039;s a good idea to [[User Jobs#Looking for unused GPUs|check for unused GPUs]] before choosing what to request. Requesting a GPU that is currently idle will help your job get executed sooner.&lt;br /&gt;
&lt;br /&gt;
= SLURM partitions =&lt;br /&gt;
&lt;br /&gt;
Partitions are another mechanism provided by SLURM to create different levels of access to system resources. Since in Mufasa 2.0 access to resources is controlled via [[#SLURM Quality of Service (QOS)|QOSes]], partitions are not very relevant. (This is one of the main differences between Mufasa 1.0 and Mufasa 2.0.)&lt;br /&gt;
&lt;br /&gt;
Note, however, that the default values for some features of SLURM jobs (e.g., duration) are [[#Default values|set by the partition]].&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, there is a single SLURM partition, called &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt;, and all jobs run on it. The state of &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt; can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sinfo -o &amp;quot;%10P %5a %9T %11L %10l&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PARTITION  AVAIL STATE     DEFAULTTIME TIMELIMIT &lt;br /&gt;
jobs*      up    idle      1:00:00     3-00:00:00&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where columns correspond to the following information:&lt;br /&gt;
&lt;br /&gt;
:; PARTITION&lt;br /&gt;
:: name of the partition; the asterisks indicates that it&amp;#039;s the default one&lt;br /&gt;
&lt;br /&gt;
:; AVAIL&lt;br /&gt;
:: state/availability of the partition: see [[#Partition availability|below]]&lt;br /&gt;
&lt;br /&gt;
:; STATE&lt;br /&gt;
:: state (using [https://slurm.schedmd.com/sinfo.html#SECTION_NODE-STATE-CODES these codes])&lt;br /&gt;
:: typical values are &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;mixed&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that some of the resources are busy executing jobs while other are idle, and &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;allocated&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that all of the resources are in use&lt;br /&gt;
&lt;br /&gt;
:; DEFAULTTIME&lt;br /&gt;
:: default runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:; TIMELIMIT&lt;br /&gt;
:: maximum runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
The asterisk at the end of the partition name indicates the default partition, i.e. the one on which jobs which do not ask for a specific partition are run.&lt;br /&gt;
&lt;br /&gt;
Command &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; does not tell you about the &amp;#039;&amp;#039;jobs&amp;#039;&amp;#039; submitted to a partition. This information is obtained, instead, with [[User Jobs#Inspecting jobs with squeue|command &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;]].&lt;br /&gt;
&lt;br /&gt;
== Partition availability ==&lt;br /&gt;
&lt;br /&gt;
The most important information that &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; provides is the &amp;#039;&amp;#039;&amp;#039;availability&amp;#039;&amp;#039;&amp;#039; (also called &amp;#039;&amp;#039;&amp;#039;state&amp;#039;&amp;#039;&amp;#039;) of partitions. This is shown in column &amp;quot;AVAIL&amp;quot;. Possible partition states are:&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is available&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed as soon as resources allow&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is in the process of becoming unavailable (i.e., of entering the &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; state: see below)&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is unavailable&lt;br /&gt;
:: There are no running jobs&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
When a partition goes from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; no harm is done to running jobs. In a normally functioning SLURM system, the passage from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; happens only when no jobs are running on the partition. If (e.g., due to a malfunction) the passage happens with jobs still running, they get killed.&lt;br /&gt;
&lt;br /&gt;
A partition in state &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; requires intervention by a [[Roles|Job Administrator]] to be restored to &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
== Default values ==&lt;br /&gt;
&lt;br /&gt;
The features of SLURM partitions, including the &amp;#039;&amp;#039;&amp;#039;default values&amp;#039;&amp;#039;&amp;#039; which are applied to jobs that do not make explicit requests, can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scontrol show partition&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PartitionName=jobs&lt;br /&gt;
   AllowGroups=ALL AllowAccounts=ALL AllowQos=nogpu,gpulight,gpu,gpuwide,gpuheavy-20,gpuheavy-40&lt;br /&gt;
   AllocNodes=ALL Default=YES QoS=N/A&lt;br /&gt;
   DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO&lt;br /&gt;
   MaxNodes=UNLIMITED MaxTime=3-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED&lt;br /&gt;
   Nodes=gn01&lt;br /&gt;
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO&lt;br /&gt;
   OverTimeLimit=NONE PreemptMode=OFF&lt;br /&gt;
   State=UP TotalCPUs=48 TotalNodes=1 SelectTypeParameters=NONE&lt;br /&gt;
   JobDefaults=(null)&lt;br /&gt;
   DefMemPerNode=4096 MaxMemPerNode=UNLIMITED&lt;br /&gt;
   TRES=cpu=48,mem=1011435M,node=1,billing=49,gres/gpu=13,gres/gpu:3g.20gb=5,gres/gpu:40gb=3,gres/gpu:4g.20gb=5&lt;br /&gt;
   TRESBillingWeights=cpu=1.0,gres/gpu:3g.20gb=6.0,gres/gpu:4g.20gb=6.0,gres/gpu:40gb=6.0,mem=0.05g&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In the output of &amp;lt;code&amp;gt;scontrol show partition&amp;lt;/code&amp;gt; the most relevant default values for Mufasa users are:&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefaultTime&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default execution time assigned to a job run on the partition (e.g., 1 hour)&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefMemPerNode&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default amount of RAM assigned to a job run on the partition (e.g., 4GB)&lt;br /&gt;
&lt;br /&gt;
= System resources subjected to limitations =&lt;br /&gt;
&lt;br /&gt;
In systems based on SLURM like Mufasa, &amp;#039;&amp;#039;&amp;#039;TRES (Trackable RESources)&amp;#039;&amp;#039;&amp;#039; are (from [https://slurm.schedmd.com/tres.html SLURM&amp;#039;s documentation] &amp;quot;&amp;#039;&amp;#039;resources that can be tracked for usage or used to enforce limits against.&amp;#039;&amp;#039;&amp;quot;&lt;br /&gt;
&lt;br /&gt;
TRES include CPUs, RAM and &amp;#039;&amp;#039;&amp;#039;GRES&amp;#039;&amp;#039;&amp;#039;. The last term stands for &amp;#039;&amp;#039;Generic RESources&amp;#039;&amp;#039; that a job may need for its execution. In Mufasa, the only &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; resources are the GPUs.&lt;br /&gt;
&lt;br /&gt;
== &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax ==&lt;br /&gt;
&lt;br /&gt;
To ask SLURM to assign GRES resources (i.e., GPUs) to a job, a special syntax must be used. Precisely, the name of each GPU resource takes the form&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:Name:Type&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
Considering the [[System#CPUs and GPUs|GPU complement of Mufasa]], &amp;lt;code&amp;gt;Type&amp;lt;/code&amp;gt; takes the following values:&lt;br /&gt;
&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:40gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 40 Gbytes of RAM&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:4g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 4 compute units&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 3 compute units&lt;br /&gt;
&lt;br /&gt;
So, for instance,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
identifies a resource corresponding to a GPU with 20 GB of RAM and 3 compute units.&lt;br /&gt;
&lt;br /&gt;
When asking for a GRES resource (e.g., in an &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command or an &amp;lt;code&amp;gt;SBATCH&amp;lt;/code&amp;gt; directive of an [[User Jobs#Using execution scripts to run jobs|execution script]]), the syntax required by SLURM is&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:&amp;lt;Type&amp;gt;:&amp;lt;Quantity&amp;gt;&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
where &amp;lt;code&amp;gt;Quantity&amp;lt;/code&amp;gt; is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of type &amp;lt;code&amp;gt;4g.20gb&amp;lt;/code&amp;gt; the syntax is&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:4g.20gb:2&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SLURM&amp;#039;s &amp;#039;&amp;#039;generic resources&amp;#039;&amp;#039; are defined in &amp;lt;code&amp;gt;/etc/slurm/gres.conf&amp;lt;/code&amp;gt;. In order to make GPUs available to SLURM&amp;#039;s &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; management, Mufasa makes use of Nvidia&amp;#039;s [https://developer.nvidia.com/nvidia-management-library-nvml NVML library]. For additional information see [https://slurm.schedmd.com/gres.html SLURM&amp;#039;s documentation].&lt;/div&gt;</summary>
		<author><name>GiulioFontana</name></author>
	</entry>
	<entry>
		<id>https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2588</id>
		<title>SLURM</title>
		<link rel="alternate" type="text/html" href="https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2588"/>
		<updated>2026-11-04T09:03:41Z</updated>

		<summary type="html">&lt;p&gt;GiulioFontana: /* Elements determining job priority */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page presents the features of SLURM that are most relevant to Mufasa&amp;#039;s [[Roles|Job Users]]. Job Users can submit jobs for execution, cancel their own jobs, and see other users&amp;#039; jobs (but not intervene on them).&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa &amp;#039;&amp;#039;&amp;#039;must use SLURM&amp;#039;&amp;#039;&amp;#039; to run resource-heavy processes, i.e. computing jobs that require one or more of the following:&lt;br /&gt;
* GPUs&lt;br /&gt;
* multiple CPUs&lt;br /&gt;
* powerful CPUs&lt;br /&gt;
* a significant amount of RAM&lt;br /&gt;
&lt;br /&gt;
In fact, only processes run via SLURM have access to all the resources of Mufasa. Processes run outside SLURM are executed by the [[System#Login server|login server]] virtual machine, which has minimal resources and no GPUs. Using SLURM is therefore the only way to execute resource-heavy jobs on Mufasa (this is a key difference between Mufasa 1.0 and Mufasa 2.0).&lt;br /&gt;
&lt;br /&gt;
= SLURM in a nutshell =&lt;br /&gt;
&lt;br /&gt;
Computation jobs on Mufasa needs to be launched via [[System#The SLURM job scheduling system|SLURM]]. SLURM provides jobs with access to the [[#System resources subjected to limitations|physical resources]] of Mufasa, such as CPUs, GPUs and RAM. Thanks to SLURM, processing jobs share system resources, optimising their occupation and availability. &lt;br /&gt;
&lt;br /&gt;
When a user runs a job, the job does not get executed immediately and is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039;. SLURM executes jobs according to their order in the queue: the top job in the queue gets executed as soon as the necessary resources are available, while jobs lower in the queue wait longer. The position of a job in the queue is due to the &amp;#039;&amp;#039;&amp;#039;[[#Job priority|priority]]&amp;#039;&amp;#039;&amp;#039; assigned to it by SLURM, with higher-priority jobs closer to the top. As a general rule:&lt;br /&gt;
&lt;br /&gt;
;: &amp;#039;&amp;#039;&amp;#039;the greater the fraction of Mufasa&amp;#039;s overall resources that a job asks for, the lower the job&amp;#039;s priority will be&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
The priority mechanism is used to encourage users to use Mufasa&amp;#039;s resources (i.e.: GPUs, CPUs, RAM, execution time) in an effective and equitable manner. This page includes a [[#How_to_maximise_the_priority_of_your_jobs|chart explaining how to maximise the priority of your jobs]].&lt;br /&gt;
&lt;br /&gt;
The &amp;#039;&amp;#039;&amp;#039;time&amp;#039;&amp;#039;&amp;#039; available to a job for its execution is controlled by SLURM. When a user requests execution of a job, they must specify the duration of the time slot that the job needs. The job must complete its execution before the end of the requested time slot, because at the end of such slot it gets killed by SLURM.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0 access to system resources is managed via SLURM&amp;#039;s [[#SLURM Quality of Service (QOS)|Quality of Service (QOS)]] mechanism (Mufasa 1.0 used [[#SLURM_partitions|partitions]] instead). While [[User Jobs#Running jobs with SLURM|launching a processing job via SLURM]], the user must always specify the chosen QOS. QOSes differ in the set of resources that they provide access to because each of them is designed to fit a given type of job.&lt;br /&gt;
&lt;br /&gt;
Mufasa sets limits to the number of jobs by the same user. This page includes a [[#Limits on jobs by the same user|table summarising such limits]].&lt;br /&gt;
&lt;br /&gt;
= SLURM Quality of Service (QOS) =&lt;br /&gt;
&lt;br /&gt;
Through &amp;#039;&amp;#039;&amp;#039;Quality of Services&amp;#039;&amp;#039;&amp;#039; (&amp;#039;&amp;#039;&amp;#039;QOSes&amp;#039;&amp;#039;&amp;#039;), SLURM lets system configurators assign a name to a set of related constraints.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, QOSes are used to define different levels of access to the server&amp;#039;s resources. When [[User Jobs|executing a job with SLURM]], a user must always &amp;#039;&amp;#039;&amp;#039;specify the QOS&amp;#039;&amp;#039;&amp;#039; that their job will use: this choice, in turn, determines what resources the job is able to access and influences the [[#Job priority|priority]] of the job.&lt;br /&gt;
&lt;br /&gt;
Mufasa&amp;#039;s QOSes and their features can be inspected with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list qos format=name%-11,priority,MaxSubmit,maxwall,maxtres%-80,mintres%-18&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following (see below for the same data in table form):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
Name          Priority MaxSubmit     MaxWall MaxTRES                                                                          MinTRES            &lt;br /&gt;
----------- ---------- --------- ----------- -------------------------------------------------------------------------------- ------------------ &lt;br /&gt;
normal               0                                                                                                                           &lt;br /&gt;
nogpu                4         1  3-00:00:00 cpu=16,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=128G                    &lt;br /&gt;
gpuheavy-20          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=2,mem=128G             gres/gpu:4g.20gb=1 &lt;br /&gt;
gpuheavy-40          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=1,gres/gpu:4g.20gb=0,mem=128G             gres/gpu:40gb=1    &lt;br /&gt;
gpulight             8         1    12:00:00 cpu=2,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpu                  2         1  1-00:00:00 cpu=8,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpuwide              2         2  1-00:00:00 cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=1,mem=64G              gres/gpu:4g.20gb=1 &lt;br /&gt;
build               32         1    02:00:00 cpu=2,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=16G&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The columns of this output are the following:&lt;br /&gt;
&lt;br /&gt;
:; Name&lt;br /&gt;
:: name of the QOS&lt;br /&gt;
&lt;br /&gt;
:; Priority&lt;br /&gt;
:: priority tier associated to the QOS (higher value = higher priority): see [[#Job priority|Job priority]] for details&lt;br /&gt;
&lt;br /&gt;
:; MaxSubmit&lt;br /&gt;
:: maximum number of jobs from a single user that can be submitted to SLURM with this QOS; submitted jobs include both running and queued jobs&lt;br /&gt;
:: See [[#Limits on jobs by the same user|Limits on jobs by the same user]] for an overview of the limits on jobs set by Mufasa.&lt;br /&gt;
&lt;br /&gt;
:; MaxWall&lt;br /&gt;
:: maximum wall clock duration of the jobs using the QOS (after which they are killed by SLURM), in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
:: For some QOSes these are not set: it means that they are determined by the [[#SLURM partitions|partition]]. Partitions also define the [[#Default values|default duration]] of jobs.&lt;br /&gt;
&lt;br /&gt;
:; MaxTRES&lt;br /&gt;
:: amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) available to a job using the QOS, where&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;cpu=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of CPUs (i.e., processor cores) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of CPUs specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;gres/&amp;#039;&amp;#039;gpu:Type&amp;#039;&amp;#039;=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of GPUs of class &amp;lt;code&amp;gt;&amp;#039;&amp;#039;Type&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; (see [[User Jobs#gres syntax|&amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax]]) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; (for QOSes that allow access to GPUs) if not specified, the job cannot be launched&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;mem=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;G&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum amount of system RAM is &amp;#039;&amp;#039;K&amp;#039;&amp;#039; GBytes&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of RAM specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
&lt;br /&gt;
:; MinTRES&lt;br /&gt;
:: minimum amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) that a job using the QOS must request in order to actually get executed by SLURM.&lt;br /&gt;
:: (If your job does not actually need these resources, you&amp;#039;ve chosen the wrong QOS: you can use one with a higher priority.)&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; QOS is the default one, and exists only to ensure that users always specify a QOS when running a job. Since &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; has zero priority and no resources, a job run using this QOS would never be run.&lt;br /&gt;
&lt;br /&gt;
The information provided by the &amp;lt;code&amp;gt;sacctmgr list qos&amp;lt;/code&amp;gt; command above is summarised by the following table:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! QOS&lt;br /&gt;
! Priority&amp;lt;/br&amp;gt;tier&lt;br /&gt;
! Max&amp;lt;/br&amp;gt;Submit&lt;br /&gt;
! MaxWall&amp;lt;/br&amp;gt;[h]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;CPUs&lt;br /&gt;
! max RAM&amp;lt;/br&amp;gt;[GB]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;3g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;4g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;40GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | build&lt;br /&gt;
| 32&lt;br /&gt;
| 1&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 16&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpulight&lt;br /&gt;
| 8&lt;br /&gt;
| 1&lt;br /&gt;
| 12&lt;br /&gt;
| 2&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | nogpu&lt;br /&gt;
| 4&lt;br /&gt;
| 1&lt;br /&gt;
| 72&lt;br /&gt;
| 16&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpu&lt;br /&gt;
| 2&lt;br /&gt;
| 1&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuwide&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-20&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| 2 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-40&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
A key piece of information in the table above is the &amp;#039;&amp;#039;&amp;#039;priority tier&amp;#039;&amp;#039;&amp;#039; associated to each QOS. In Mufasa 2.0, priority tiers are used to encourage users to use the &amp;#039;&amp;#039;&amp;#039;least powerful QOS that is compatible with their needs&amp;#039;&amp;#039;&amp;#039;, where &amp;quot;powerful&amp;quot; means &amp;quot;rich with resources&amp;quot;. Less powerful QOSes increase the priority of the jobs that use them, so these jobs tend to be executed sooner. See [[#Job priority|Job priority]] for details about how priority affects the execution order of jobs in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
== The &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS ==&lt;br /&gt;
&lt;br /&gt;
This QOS is specifically designed to be used by Mufasa users to &amp;#039;&amp;#039;&amp;#039;quickly build [[System#Containers|container images]]&amp;#039;&amp;#039;&amp;#039;. Its associated priority tier is very high: this, combined with the fact that jobs using this QOS require few resources, means that such jobs usually get executed very soon.&lt;br /&gt;
&lt;br /&gt;
On the other hand, the limited resources and the lack of access to the GPUs make the &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS unsuitable for other tasks.&lt;br /&gt;
&lt;br /&gt;
See [[Singularity#Building Singularity images|Building Singularity images]] for directions about building Singularity container images.&lt;br /&gt;
&lt;br /&gt;
== Restricted QOSes ==&lt;br /&gt;
&lt;br /&gt;
In Mufasa, the most powerful QOSes are reserved to researchers (called &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users: these include academic personnel and Ph.D. students). M.Sc. students, called &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users, cannot use them while running jobs.&lt;br /&gt;
&lt;br /&gt;
See [[#research users and students users|below]] to understand how &amp;#039;&amp;#039;user categories&amp;#039;&amp;#039; work in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
= &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users =&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa belong to two &amp;#039;&amp;#039;&amp;#039;user categories&amp;#039;&amp;#039;&amp;#039;. Categories provide users with different access to  to Mufasa&amp;#039;s resources: the idea being to provide researchers with more access while still letting students benefit from the server&amp;#039;s capabilities.&lt;br /&gt;
&lt;br /&gt;
User categories in Mufasa are the following:&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. academic personnel and Ph.D. students&lt;br /&gt;
::: * can use all QOSes, including the the [[#Restricted QOSes|restricted ones]]&lt;br /&gt;
::: * their jobs have a higher &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a higher number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. M.Sc. students&lt;br /&gt;
::: * cannot use the [[#Restricted QOSes|restricted QOSes]]&lt;br /&gt;
::: * their jobs have a lower &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a lower number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
You can inspect the differences in priority and running jobs between &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association format=&amp;quot;account,priority,maxjobs&amp;quot; | grep -E &amp;#039;Account|research|students&amp;#039;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
   Account   Priority MaxJobs &lt;br /&gt;
  research          4       2 &lt;br /&gt;
  students          1       1&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
To know what limits apply to your own user, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association where user=$USER format=&amp;quot;user,priority,maxjobs,qos%-60&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
      User   Priority MaxJobs QOS                                                          &lt;br /&gt;
---------- ---------- ------- ------------------------------------------------------------ &lt;br /&gt;
    preali          4       2 build,gpu,gpuheavy-20,gpuheavy-40,gpulight,gpuwide,nogpu&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The list under &amp;quot;QOS&amp;quot; shows what QOSes your user is allowed to use when [[User Jobs|running jobs]]. &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users can use all of them, while &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users can only access a subset of them.&lt;br /&gt;
&lt;br /&gt;
= Limits on jobs by the same user =&lt;br /&gt;
&lt;br /&gt;
Mufasa enforces limits on the number of jobs from a single user. Such limits aim at preventing users from &amp;quot;hogging&amp;quot; system resources, and apply to:&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;submitted jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that the user asked SLURM to execute, each of which may currently be either running or queued&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;running jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that are currently in execution&lt;br /&gt;
&lt;br /&gt;
Limits on submitted jobs are set via [[#SLURM Quality of Service (QOS)|QOSes]], while limits on running jobs are set via [[#research users and students users|user category]].&lt;br /&gt;
&lt;br /&gt;
The following table summarises what limits exist:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! &lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;running&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;submitted&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | global limits&amp;lt;br/&amp;gt;(system-wide)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to limits set by individual QOSes (below)&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | limits for&amp;lt;br/&amp;gt;individual&amp;lt;/br&amp;gt;QOSes&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to global limits (above)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for the&amp;#039;&amp;#039; &amp;lt;code&amp;gt;gpuwide&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;QOS&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for each of the other QOSes&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
= Job priority =&lt;br /&gt;
&lt;br /&gt;
Once the execution of a job has been requested, the job is not run immediately: it is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039; by SLURM, together with all the other jobs awaiting execution. The job on top of the queue at any time is the first to be put into execution as soon as the resources it requires are available. &lt;br /&gt;
&lt;br /&gt;
The order of the items in the job queue depends on their &amp;#039;&amp;#039;&amp;#039;priority&amp;#039;&amp;#039;&amp;#039;. The job with the highest priority is put by SLURM on top of the queue, while all other jobs are put in the queue in order of descending priority.&lt;br /&gt;
&lt;br /&gt;
The goal of SLURM is to maximise resource availability: i.e., to ensure the shorter possible wait time before job execution. To achieve this goal, in Mufasa SLURM is configured to &amp;#039;&amp;#039;&amp;#039;encourage users not to ask for resources or execution time that their job doesn&amp;#039;t need&amp;#039;&amp;#039;&amp;#039;. This is done via the priority mechanism: the more resources and/or time a job requests, the lower its priority will be; and the later it will be executed.&lt;br /&gt;
&lt;br /&gt;
Priority management in Mufasa is designed to set up a &amp;#039;&amp;#039;&amp;#039;virtuous cycle&amp;#039;&amp;#039;&amp;#039; where users, by carefully choosing what to ask for, obtain two results:&lt;br /&gt;
* they ensure that their job is executed as soon as possible;&lt;br /&gt;
* they leave as much as possible of Mufasa&amp;#039;s resources free for other users&amp;#039;s jobs.&lt;br /&gt;
&lt;br /&gt;
== Elements determining job priority ==&lt;br /&gt;
In Mufasa, the priority of a job is computed by SLURM according to the following elements:&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#research users and students users|User category]]&amp;#039;&amp;#039;&amp;#039; (i.e., &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;)&lt;br /&gt;
::: Used to provide higher priority to jobs run by &amp;#039;&amp;#039;&amp;#039;research personnel&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#SLURM Quality of Service (QOS)|QOS]]&amp;#039;&amp;#039;&amp;#039; used by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;less resources&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Number of CPUs&amp;#039;&amp;#039;&amp;#039; requested by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;a lower number of CPUs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job Age&amp;#039;&amp;#039;&amp;#039;, i.e. the time that the job has been waiting in the queue&lt;br /&gt;
::: Used to provide higher priority to jobs which have been &amp;#039;&amp;#039;&amp;#039;queued for a longer time&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;FairShare&amp;#039;&amp;#039;&amp;#039;, i.e. a factor computed by SLURM to balance use of the system by different users&lt;br /&gt;
::: Used to provide higher priority to jobs by users who &amp;#039;&amp;#039;&amp;#039;used Mufasa&amp;#039;s resources less than others&amp;#039;&amp;#039;&amp;#039; (see [[#How FairShare works|below]])&lt;br /&gt;
&lt;br /&gt;
You can understand how each of these elements contributes to the priority of the jobs waiting for execution (both yours and of other users) with command &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;. A tailored version of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; that only displays the parameters listed above is the following:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
prio -o &amp;quot;%6i %9u %.8Y %.10B %.5Q %.9T %.5A %.5F&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Things to consider when using &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;&lt;br /&gt;
:: Its output is only relevant for jobs &amp;#039;&amp;#039;&amp;#039;not yet in execution&amp;#039;&amp;#039;&amp;#039;. For an already running job, the values shown by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; correspond to the priority that the job would get if evaluated now, which generally differs from its actual priority.&lt;br /&gt;
:: Jobs listed by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; may temporarily differ from jobs listed by &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;: this is due to the fact that SLURM updates its internal data with a periodicity of a few minutes.&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Here is an example of the output of the command above:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
JOBID  USER      PRIORITY      ASSOC   QOS      TRES   AGE FAIRS&lt;br /&gt;
5774   itombini      3576       2500   750  cpu=-800   990   136&lt;br /&gt;
5780   fbottazzi     2350        625   750  cpu=-600   504  1071&lt;br /&gt;
5783   mzoccola      1849        625   750  cpu=-800   300   974&lt;br /&gt;
5790   amoglia       2801       2500   375  cpu=-800   239   487&lt;br /&gt;
5791   qliu          2315       2500   375  cpu=-800   221    19&lt;br /&gt;
5795   lpassoni      2600       2500   750  cpu=-800    92    58&lt;br /&gt;
5805   adurat        3154       2500   750  cpu=-800    62   643&lt;br /&gt;
5806   itombini      2645       2500   750  cpu=-800    59   136&lt;br /&gt;
5813   fpastori      5385       2500  3000  cpu=-200    47    39&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
In it, column PRIORITY provides the overall priority of each job. Each priority value is the sum of the five values to its right; in particular, column ASSOC corresponds to user category, while column TRES corresponds to the number of CPUs. Note that the latter contribution to priority is &amp;#039;&amp;#039;negative&amp;#039;&amp;#039;, i.e. it is a penalty.&lt;br /&gt;
&lt;br /&gt;
=== How FairShare works ===&lt;br /&gt;
In Mufasa 2.0, the FairShare of each user is computed according to these rules:&lt;br /&gt;
# It considers the quantity used by (current and past) user jobs of:&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;CPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of CPUs used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;RAM&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the amount of RAM used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;GPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of GPUs used&lt;br /&gt;
#** note: 40 GB GPUs have double the impact on FairShare than 20 GB GPUs&lt;br /&gt;
# The impact on FairShare of any use of resources is &amp;#039;&amp;#039;&amp;#039;proportional to its duration&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
#* example: using 32 GB of RAM for 48h has the same impact of using 64 GB of RAM for 24h&lt;br /&gt;
# FairShare has a &amp;quot;fading memory&amp;quot;: i.e., resource use has less and less impact on FairShare the farther in the past it is&lt;br /&gt;
#* consequence: over time, the &amp;quot;history&amp;quot; of a user gets forgotten by FairShare&lt;br /&gt;
&lt;br /&gt;
== How to maximise the priority of your jobs ==&lt;br /&gt;
&lt;br /&gt;
Every time you run a SLURM job, follow these guidelines:&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Choose the less powerful QOS compatible with the needs of your job&lt;br /&gt;
:: QOSes with access to less resources have higher base priority&lt;br /&gt;
&lt;br /&gt;
; Only request CPUs and RAM that your job really needs&lt;br /&gt;
:: Usually, code exploits multiple CPUs only if specifically designed to do so: if unsure, only ask for 1 CPU&lt;br /&gt;
:: The fewer resources your job asks for, the less it will wait before they become available&lt;br /&gt;
:: The more CPUs your job asks for, the stronger the penalty to its priority&lt;br /&gt;
:: Asking for fewer CPUs and/or less RAM improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Do not request more time than your jobs needs to complete&lt;br /&gt;
:: Make a worst-case estimate and only ask for that duration&lt;br /&gt;
:: Asking for less of Mufasa&amp;#039;s time improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Debug and test code using less powerful QOSes before running it with more powerful QOSes&lt;br /&gt;
:: Your test jobs will get a higher priority and your FairShare will improve&lt;br /&gt;
&lt;br /&gt;
; Cancel jobs that you don&amp;#039;t need them anymore&lt;br /&gt;
:: Always use [[User_Jobs#Cancelling_a_job_with_scancel|scancel]] to delete completed (or crashed) jobs: your Fairshare will improve&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Protip: it&amp;#039;s a good idea to [[User Jobs#Looking for unused GPUs|check for unused GPUs]] before choosing what to request. Requesting a GPU that is currently idle will help your job get executed sooner.&lt;br /&gt;
&lt;br /&gt;
= SLURM partitions =&lt;br /&gt;
&lt;br /&gt;
Partitions are another mechanism provided by SLURM to create different levels of access to system resources. Since in Mufasa 2.0 access to resources is controlled via [[#SLURM Quality of Service (QOS)|QOSes]], partitions are not very relevant. (This is one of the main differences between Mufasa 1.0 and Mufasa 2.0.)&lt;br /&gt;
&lt;br /&gt;
Note, however, that the default values for some features of SLURM jobs (e.g., duration) are [[#Default values|set by the partition]].&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, there is a single SLURM partition, called &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt;, and all jobs run on it. The state of &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt; can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sinfo -o &amp;quot;%10P %5a %9T %11L %10l&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PARTITION  AVAIL STATE     DEFAULTTIME TIMELIMIT &lt;br /&gt;
jobs*      up    idle      1:00:00     3-00:00:00&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where columns correspond to the following information:&lt;br /&gt;
&lt;br /&gt;
:; PARTITION&lt;br /&gt;
:: name of the partition; the asterisks indicates that it&amp;#039;s the default one&lt;br /&gt;
&lt;br /&gt;
:; AVAIL&lt;br /&gt;
:: state/availability of the partition: see [[#Partition availability|below]]&lt;br /&gt;
&lt;br /&gt;
:; STATE&lt;br /&gt;
:: state (using [https://slurm.schedmd.com/sinfo.html#SECTION_NODE-STATE-CODES these codes])&lt;br /&gt;
:: typical values are &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;mixed&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that some of the resources are busy executing jobs while other are idle, and &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;allocated&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that all of the resources are in use&lt;br /&gt;
&lt;br /&gt;
:; DEFAULTTIME&lt;br /&gt;
:: default runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:; TIMELIMIT&lt;br /&gt;
:: maximum runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
The asterisk at the end of the partition name indicates the default partition, i.e. the one on which jobs which do not ask for a specific partition are run.&lt;br /&gt;
&lt;br /&gt;
Command &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; does not tell you about the &amp;#039;&amp;#039;jobs&amp;#039;&amp;#039; submitted to a partition. This information is obtained, instead, with [[User Jobs#Inspecting jobs with squeue|command &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;]].&lt;br /&gt;
&lt;br /&gt;
== Partition availability ==&lt;br /&gt;
&lt;br /&gt;
The most important information that &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; provides is the &amp;#039;&amp;#039;&amp;#039;availability&amp;#039;&amp;#039;&amp;#039; (also called &amp;#039;&amp;#039;&amp;#039;state&amp;#039;&amp;#039;&amp;#039;) of partitions. This is shown in column &amp;quot;AVAIL&amp;quot;. Possible partition states are:&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is available&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed as soon as resources allow&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is in the process of becoming unavailable (i.e., of entering the &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; state: see below)&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is unavailable&lt;br /&gt;
:: There are no running jobs&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
When a partition goes from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; no harm is done to running jobs. In a normally functioning SLURM system, the passage from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; happens only when no jobs are running on the partition. If (e.g., due to a malfunction) the passage happens with jobs still running, they get killed.&lt;br /&gt;
&lt;br /&gt;
A partition in state &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; requires intervention by a [[Roles|Job Administrator]] to be restored to &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
== Default values ==&lt;br /&gt;
&lt;br /&gt;
The features of SLURM partitions, including the &amp;#039;&amp;#039;&amp;#039;default values&amp;#039;&amp;#039;&amp;#039; which are applied to jobs that do not make explicit requests, can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scontrol show partition&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PartitionName=jobs&lt;br /&gt;
   AllowGroups=ALL AllowAccounts=ALL AllowQos=nogpu,gpulight,gpu,gpuwide,gpuheavy-20,gpuheavy-40&lt;br /&gt;
   AllocNodes=ALL Default=YES QoS=N/A&lt;br /&gt;
   DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO&lt;br /&gt;
   MaxNodes=UNLIMITED MaxTime=3-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED&lt;br /&gt;
   Nodes=gn01&lt;br /&gt;
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO&lt;br /&gt;
   OverTimeLimit=NONE PreemptMode=OFF&lt;br /&gt;
   State=UP TotalCPUs=48 TotalNodes=1 SelectTypeParameters=NONE&lt;br /&gt;
   JobDefaults=(null)&lt;br /&gt;
   DefMemPerNode=4096 MaxMemPerNode=UNLIMITED&lt;br /&gt;
   TRES=cpu=48,mem=1011435M,node=1,billing=49,gres/gpu=13,gres/gpu:3g.20gb=5,gres/gpu:40gb=3,gres/gpu:4g.20gb=5&lt;br /&gt;
   TRESBillingWeights=cpu=1.0,gres/gpu:3g.20gb=6.0,gres/gpu:4g.20gb=6.0,gres/gpu:40gb=6.0,mem=0.05g&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In the output of &amp;lt;code&amp;gt;scontrol show partition&amp;lt;/code&amp;gt; the most relevant default values for Mufasa users are:&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefaultTime&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default execution time assigned to a job run on the partition (e.g., 1 hour)&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefMemPerNode&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default amount of RAM assigned to a job run on the partition (e.g., 4GB)&lt;br /&gt;
&lt;br /&gt;
= System resources subjected to limitations =&lt;br /&gt;
&lt;br /&gt;
In systems based on SLURM like Mufasa, &amp;#039;&amp;#039;&amp;#039;TRES (Trackable RESources)&amp;#039;&amp;#039;&amp;#039; are (from [https://slurm.schedmd.com/tres.html SLURM&amp;#039;s documentation] &amp;quot;&amp;#039;&amp;#039;resources that can be tracked for usage or used to enforce limits against.&amp;#039;&amp;#039;&amp;quot;&lt;br /&gt;
&lt;br /&gt;
TRES include CPUs, RAM and &amp;#039;&amp;#039;&amp;#039;GRES&amp;#039;&amp;#039;&amp;#039;. The last term stands for &amp;#039;&amp;#039;Generic RESources&amp;#039;&amp;#039; that a job may need for its execution. In Mufasa, the only &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; resources are the GPUs.&lt;br /&gt;
&lt;br /&gt;
== &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax ==&lt;br /&gt;
&lt;br /&gt;
To ask SLURM to assign GRES resources (i.e., GPUs) to a job, a special syntax must be used. Precisely, the name of each GPU resource takes the form&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:Name:Type&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
Considering the [[System#CPUs and GPUs|GPU complement of Mufasa]], &amp;lt;code&amp;gt;Type&amp;lt;/code&amp;gt; takes the following values:&lt;br /&gt;
&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:40gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 40 Gbytes of RAM&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:4g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 4 compute units&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 3 compute units&lt;br /&gt;
&lt;br /&gt;
So, for instance,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
identifies a resource corresponding to a GPU with 20 GB of RAM and 3 compute units.&lt;br /&gt;
&lt;br /&gt;
When asking for a GRES resource (e.g., in an &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command or an &amp;lt;code&amp;gt;SBATCH&amp;lt;/code&amp;gt; directive of an [[User Jobs#Using execution scripts to run jobs|execution script]]), the syntax required by SLURM is&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:&amp;lt;Type&amp;gt;:&amp;lt;Quantity&amp;gt;&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
where &amp;lt;code&amp;gt;Quantity&amp;lt;/code&amp;gt; is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of type &amp;lt;code&amp;gt;4g.20gb&amp;lt;/code&amp;gt; the syntax is&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:4g.20gb:2&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SLURM&amp;#039;s &amp;#039;&amp;#039;generic resources&amp;#039;&amp;#039; are defined in &amp;lt;code&amp;gt;/etc/slurm/gres.conf&amp;lt;/code&amp;gt;. In order to make GPUs available to SLURM&amp;#039;s &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; management, Mufasa makes use of Nvidia&amp;#039;s [https://developer.nvidia.com/nvidia-management-library-nvml NVML library]. For additional information see [https://slurm.schedmd.com/gres.html SLURM&amp;#039;s documentation].&lt;/div&gt;</summary>
		<author><name>GiulioFontana</name></author>
	</entry>
	<entry>
		<id>https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2587</id>
		<title>SLURM</title>
		<link rel="alternate" type="text/html" href="https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2587"/>
		<updated>2026-11-03T16:27:01Z</updated>

		<summary type="html">&lt;p&gt;GiulioFontana: /* Elements determining job priority */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page presents the features of SLURM that are most relevant to Mufasa&amp;#039;s [[Roles|Job Users]]. Job Users can submit jobs for execution, cancel their own jobs, and see other users&amp;#039; jobs (but not intervene on them).&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa &amp;#039;&amp;#039;&amp;#039;must use SLURM&amp;#039;&amp;#039;&amp;#039; to run resource-heavy processes, i.e. computing jobs that require one or more of the following:&lt;br /&gt;
* GPUs&lt;br /&gt;
* multiple CPUs&lt;br /&gt;
* powerful CPUs&lt;br /&gt;
* a significant amount of RAM&lt;br /&gt;
&lt;br /&gt;
In fact, only processes run via SLURM have access to all the resources of Mufasa. Processes run outside SLURM are executed by the [[System#Login server|login server]] virtual machine, which has minimal resources and no GPUs. Using SLURM is therefore the only way to execute resource-heavy jobs on Mufasa (this is a key difference between Mufasa 1.0 and Mufasa 2.0).&lt;br /&gt;
&lt;br /&gt;
= SLURM in a nutshell =&lt;br /&gt;
&lt;br /&gt;
Computation jobs on Mufasa needs to be launched via [[System#The SLURM job scheduling system|SLURM]]. SLURM provides jobs with access to the [[#System resources subjected to limitations|physical resources]] of Mufasa, such as CPUs, GPUs and RAM. Thanks to SLURM, processing jobs share system resources, optimising their occupation and availability. &lt;br /&gt;
&lt;br /&gt;
When a user runs a job, the job does not get executed immediately and is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039;. SLURM executes jobs according to their order in the queue: the top job in the queue gets executed as soon as the necessary resources are available, while jobs lower in the queue wait longer. The position of a job in the queue is due to the &amp;#039;&amp;#039;&amp;#039;[[#Job priority|priority]]&amp;#039;&amp;#039;&amp;#039; assigned to it by SLURM, with higher-priority jobs closer to the top. As a general rule:&lt;br /&gt;
&lt;br /&gt;
;: &amp;#039;&amp;#039;&amp;#039;the greater the fraction of Mufasa&amp;#039;s overall resources that a job asks for, the lower the job&amp;#039;s priority will be&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
The priority mechanism is used to encourage users to use Mufasa&amp;#039;s resources (i.e.: GPUs, CPUs, RAM, execution time) in an effective and equitable manner. This page includes a [[#How_to_maximise_the_priority_of_your_jobs|chart explaining how to maximise the priority of your jobs]].&lt;br /&gt;
&lt;br /&gt;
The &amp;#039;&amp;#039;&amp;#039;time&amp;#039;&amp;#039;&amp;#039; available to a job for its execution is controlled by SLURM. When a user requests execution of a job, they must specify the duration of the time slot that the job needs. The job must complete its execution before the end of the requested time slot, because at the end of such slot it gets killed by SLURM.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0 access to system resources is managed via SLURM&amp;#039;s [[#SLURM Quality of Service (QOS)|Quality of Service (QOS)]] mechanism (Mufasa 1.0 used [[#SLURM_partitions|partitions]] instead). While [[User Jobs#Running jobs with SLURM|launching a processing job via SLURM]], the user must always specify the chosen QOS. QOSes differ in the set of resources that they provide access to because each of them is designed to fit a given type of job.&lt;br /&gt;
&lt;br /&gt;
Mufasa sets limits to the number of jobs by the same user. This page includes a [[#Limits on jobs by the same user|table summarising such limits]].&lt;br /&gt;
&lt;br /&gt;
= SLURM Quality of Service (QOS) =&lt;br /&gt;
&lt;br /&gt;
Through &amp;#039;&amp;#039;&amp;#039;Quality of Services&amp;#039;&amp;#039;&amp;#039; (&amp;#039;&amp;#039;&amp;#039;QOSes&amp;#039;&amp;#039;&amp;#039;), SLURM lets system configurators assign a name to a set of related constraints.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, QOSes are used to define different levels of access to the server&amp;#039;s resources. When [[User Jobs|executing a job with SLURM]], a user must always &amp;#039;&amp;#039;&amp;#039;specify the QOS&amp;#039;&amp;#039;&amp;#039; that their job will use: this choice, in turn, determines what resources the job is able to access and influences the [[#Job priority|priority]] of the job.&lt;br /&gt;
&lt;br /&gt;
Mufasa&amp;#039;s QOSes and their features can be inspected with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list qos format=name%-11,priority,MaxSubmit,maxwall,maxtres%-80,mintres%-18&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following (see below for the same data in table form):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
Name          Priority MaxSubmit     MaxWall MaxTRES                                                                          MinTRES            &lt;br /&gt;
----------- ---------- --------- ----------- -------------------------------------------------------------------------------- ------------------ &lt;br /&gt;
normal               0                                                                                                                           &lt;br /&gt;
nogpu                4         1  3-00:00:00 cpu=16,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=128G                    &lt;br /&gt;
gpuheavy-20          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=2,mem=128G             gres/gpu:4g.20gb=1 &lt;br /&gt;
gpuheavy-40          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=1,gres/gpu:4g.20gb=0,mem=128G             gres/gpu:40gb=1    &lt;br /&gt;
gpulight             8         1    12:00:00 cpu=2,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpu                  2         1  1-00:00:00 cpu=8,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpuwide              2         2  1-00:00:00 cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=1,mem=64G              gres/gpu:4g.20gb=1 &lt;br /&gt;
build               32         1    02:00:00 cpu=2,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=16G&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The columns of this output are the following:&lt;br /&gt;
&lt;br /&gt;
:; Name&lt;br /&gt;
:: name of the QOS&lt;br /&gt;
&lt;br /&gt;
:; Priority&lt;br /&gt;
:: priority tier associated to the QOS (higher value = higher priority): see [[#Job priority|Job priority]] for details&lt;br /&gt;
&lt;br /&gt;
:; MaxSubmit&lt;br /&gt;
:: maximum number of jobs from a single user that can be submitted to SLURM with this QOS; submitted jobs include both running and queued jobs&lt;br /&gt;
:: See [[#Limits on jobs by the same user|Limits on jobs by the same user]] for an overview of the limits on jobs set by Mufasa.&lt;br /&gt;
&lt;br /&gt;
:; MaxWall&lt;br /&gt;
:: maximum wall clock duration of the jobs using the QOS (after which they are killed by SLURM), in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
:: For some QOSes these are not set: it means that they are determined by the [[#SLURM partitions|partition]]. Partitions also define the [[#Default values|default duration]] of jobs.&lt;br /&gt;
&lt;br /&gt;
:; MaxTRES&lt;br /&gt;
:: amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) available to a job using the QOS, where&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;cpu=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of CPUs (i.e., processor cores) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of CPUs specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;gres/&amp;#039;&amp;#039;gpu:Type&amp;#039;&amp;#039;=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of GPUs of class &amp;lt;code&amp;gt;&amp;#039;&amp;#039;Type&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; (see [[User Jobs#gres syntax|&amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax]]) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; (for QOSes that allow access to GPUs) if not specified, the job cannot be launched&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;mem=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;G&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum amount of system RAM is &amp;#039;&amp;#039;K&amp;#039;&amp;#039; GBytes&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of RAM specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
&lt;br /&gt;
:; MinTRES&lt;br /&gt;
:: minimum amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) that a job using the QOS must request in order to actually get executed by SLURM.&lt;br /&gt;
:: (If your job does not actually need these resources, you&amp;#039;ve chosen the wrong QOS: you can use one with a higher priority.)&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; QOS is the default one, and exists only to ensure that users always specify a QOS when running a job. Since &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; has zero priority and no resources, a job run using this QOS would never be run.&lt;br /&gt;
&lt;br /&gt;
The information provided by the &amp;lt;code&amp;gt;sacctmgr list qos&amp;lt;/code&amp;gt; command above is summarised by the following table:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! QOS&lt;br /&gt;
! Priority&amp;lt;/br&amp;gt;tier&lt;br /&gt;
! Max&amp;lt;/br&amp;gt;Submit&lt;br /&gt;
! MaxWall&amp;lt;/br&amp;gt;[h]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;CPUs&lt;br /&gt;
! max RAM&amp;lt;/br&amp;gt;[GB]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;3g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;4g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;40GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | build&lt;br /&gt;
| 32&lt;br /&gt;
| 1&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 16&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpulight&lt;br /&gt;
| 8&lt;br /&gt;
| 1&lt;br /&gt;
| 12&lt;br /&gt;
| 2&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | nogpu&lt;br /&gt;
| 4&lt;br /&gt;
| 1&lt;br /&gt;
| 72&lt;br /&gt;
| 16&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpu&lt;br /&gt;
| 2&lt;br /&gt;
| 1&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuwide&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-20&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| 2 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-40&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
A key piece of information in the table above is the &amp;#039;&amp;#039;&amp;#039;priority tier&amp;#039;&amp;#039;&amp;#039; associated to each QOS. In Mufasa 2.0, priority tiers are used to encourage users to use the &amp;#039;&amp;#039;&amp;#039;least powerful QOS that is compatible with their needs&amp;#039;&amp;#039;&amp;#039;, where &amp;quot;powerful&amp;quot; means &amp;quot;rich with resources&amp;quot;. Less powerful QOSes increase the priority of the jobs that use them, so these jobs tend to be executed sooner. See [[#Job priority|Job priority]] for details about how priority affects the execution order of jobs in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
== The &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS ==&lt;br /&gt;
&lt;br /&gt;
This QOS is specifically designed to be used by Mufasa users to &amp;#039;&amp;#039;&amp;#039;quickly build [[System#Containers|container images]]&amp;#039;&amp;#039;&amp;#039;. Its associated priority tier is very high: this, combined with the fact that jobs using this QOS require few resources, means that such jobs usually get executed very soon.&lt;br /&gt;
&lt;br /&gt;
On the other hand, the limited resources and the lack of access to the GPUs make the &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS unsuitable for other tasks.&lt;br /&gt;
&lt;br /&gt;
See [[Singularity#Building Singularity images|Building Singularity images]] for directions about building Singularity container images.&lt;br /&gt;
&lt;br /&gt;
== Restricted QOSes ==&lt;br /&gt;
&lt;br /&gt;
In Mufasa, the most powerful QOSes are reserved to researchers (called &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users: these include academic personnel and Ph.D. students). M.Sc. students, called &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users, cannot use them while running jobs.&lt;br /&gt;
&lt;br /&gt;
See [[#research users and students users|below]] to understand how &amp;#039;&amp;#039;user categories&amp;#039;&amp;#039; work in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
= &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users =&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa belong to two &amp;#039;&amp;#039;&amp;#039;user categories&amp;#039;&amp;#039;&amp;#039;. Categories provide users with different access to  to Mufasa&amp;#039;s resources: the idea being to provide researchers with more access while still letting students benefit from the server&amp;#039;s capabilities.&lt;br /&gt;
&lt;br /&gt;
User categories in Mufasa are the following:&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. academic personnel and Ph.D. students&lt;br /&gt;
::: * can use all QOSes, including the the [[#Restricted QOSes|restricted ones]]&lt;br /&gt;
::: * their jobs have a higher &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a higher number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. M.Sc. students&lt;br /&gt;
::: * cannot use the [[#Restricted QOSes|restricted QOSes]]&lt;br /&gt;
::: * their jobs have a lower &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a lower number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
You can inspect the differences in priority and running jobs between &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association format=&amp;quot;account,priority,maxjobs&amp;quot; | grep -E &amp;#039;Account|research|students&amp;#039;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
   Account   Priority MaxJobs &lt;br /&gt;
  research          4       2 &lt;br /&gt;
  students          1       1&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
To know what limits apply to your own user, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association where user=$USER format=&amp;quot;user,priority,maxjobs,qos%-60&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
      User   Priority MaxJobs QOS                                                          &lt;br /&gt;
---------- ---------- ------- ------------------------------------------------------------ &lt;br /&gt;
    preali          4       2 build,gpu,gpuheavy-20,gpuheavy-40,gpulight,gpuwide,nogpu&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The list under &amp;quot;QOS&amp;quot; shows what QOSes your user is allowed to use when [[User Jobs|running jobs]]. &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users can use all of them, while &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users can only access a subset of them.&lt;br /&gt;
&lt;br /&gt;
= Limits on jobs by the same user =&lt;br /&gt;
&lt;br /&gt;
Mufasa enforces limits on the number of jobs from a single user. Such limits aim at preventing users from &amp;quot;hogging&amp;quot; system resources, and apply to:&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;submitted jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that the user asked SLURM to execute, each of which may currently be either running or queued&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;running jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that are currently in execution&lt;br /&gt;
&lt;br /&gt;
Limits on submitted jobs are set via [[#SLURM Quality of Service (QOS)|QOSes]], while limits on running jobs are set via [[#research users and students users|user category]].&lt;br /&gt;
&lt;br /&gt;
The following table summarises what limits exist:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! &lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;running&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;submitted&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | global limits&amp;lt;br/&amp;gt;(system-wide)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to limits set by individual QOSes (below)&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | limits for&amp;lt;br/&amp;gt;individual&amp;lt;/br&amp;gt;QOSes&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to global limits (above)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for the&amp;#039;&amp;#039; &amp;lt;code&amp;gt;gpuwide&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;QOS&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for each of the other QOSes&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
= Job priority =&lt;br /&gt;
&lt;br /&gt;
Once the execution of a job has been requested, the job is not run immediately: it is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039; by SLURM, together with all the other jobs awaiting execution. The job on top of the queue at any time is the first to be put into execution as soon as the resources it requires are available. &lt;br /&gt;
&lt;br /&gt;
The order of the items in the job queue depends on their &amp;#039;&amp;#039;&amp;#039;priority&amp;#039;&amp;#039;&amp;#039;. The job with the highest priority is put by SLURM on top of the queue, while all other jobs are put in the queue in order of descending priority.&lt;br /&gt;
&lt;br /&gt;
The goal of SLURM is to maximise resource availability: i.e., to ensure the shorter possible wait time before job execution. To achieve this goal, in Mufasa SLURM is configured to &amp;#039;&amp;#039;&amp;#039;encourage users not to ask for resources or execution time that their job doesn&amp;#039;t need&amp;#039;&amp;#039;&amp;#039;. This is done via the priority mechanism: the more resources and/or time a job requests, the lower its priority will be; and the later it will be executed.&lt;br /&gt;
&lt;br /&gt;
Priority management in Mufasa is designed to set up a &amp;#039;&amp;#039;&amp;#039;virtuous cycle&amp;#039;&amp;#039;&amp;#039; where users, by carefully choosing what to ask for, obtain two results:&lt;br /&gt;
* they ensure that their job is executed as soon as possible;&lt;br /&gt;
* they leave as much as possible of Mufasa&amp;#039;s resources free for other users&amp;#039;s jobs.&lt;br /&gt;
&lt;br /&gt;
== Elements determining job priority ==&lt;br /&gt;
In Mufasa, the priority of a job is computed by SLURM according to the following elements:&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#research users and students users|User category]]&amp;#039;&amp;#039;&amp;#039; (i.e., &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;)&lt;br /&gt;
::: Used to provide higher priority to jobs run by &amp;#039;&amp;#039;&amp;#039;research personnel&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#SLURM Quality of Service (QOS)|QOS]]&amp;#039;&amp;#039;&amp;#039; used by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;less resources&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Number of CPUs&amp;#039;&amp;#039;&amp;#039; requested by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;a lower number of CPUs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job Age&amp;#039;&amp;#039;&amp;#039;, i.e. the time that the job has been waiting in the queue&lt;br /&gt;
::: Used to provide higher priority to jobs which have been &amp;#039;&amp;#039;&amp;#039;queued for a longer time&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;FairShare&amp;#039;&amp;#039;&amp;#039;, i.e. a factor computed by SLURM to balance use of the system by different users&lt;br /&gt;
::: Used to provide higher priority to jobs by users who &amp;#039;&amp;#039;&amp;#039;used Mufasa&amp;#039;s resources less than others&amp;#039;&amp;#039;&amp;#039; (see [[#How FairShare works|below]])&lt;br /&gt;
&lt;br /&gt;
You can understand how each of these elements contributes to the priority of the jobs waiting for execution (both yours and of other users) with command &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;. A tailored version of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; that only displays the parameters listed above is the following:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
prio -o &amp;quot;%6i %9u %.8Y %.10B %.5Q %.9T %.5A %.5F&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Important!&lt;br /&gt;
:: The output of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; is only relevant for jobs &amp;#039;&amp;#039;&amp;#039;not yet in execution&amp;#039;&amp;#039;&amp;#039;. For an already running job, the values shown by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; correspond to the priority that the job would get if computed now, which generally differs from the actual priority.&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Here is an example of the output of the command above:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
JOBID  USER      PRIORITY      ASSOC   QOS      TRES   AGE FAIRS&lt;br /&gt;
5774   itombini      3576       2500   750  cpu=-800   990   136&lt;br /&gt;
5780   fbottazzi     2350        625   750  cpu=-600   504  1071&lt;br /&gt;
5783   mzoccola      1849        625   750  cpu=-800   300   974&lt;br /&gt;
5790   amoglia       2801       2500   375  cpu=-800   239   487&lt;br /&gt;
5791   qliu          2315       2500   375  cpu=-800   221    19&lt;br /&gt;
5795   lpassoni      2600       2500   750  cpu=-800    92    58&lt;br /&gt;
5805   adurat        3154       2500   750  cpu=-800    62   643&lt;br /&gt;
5806   itombini      2645       2500   750  cpu=-800    59   136&lt;br /&gt;
5813   fpastori      5385       2500  3000  cpu=-200    47    39&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
In it, column PRIORITY provides the overall priority of each job. Each priority value is the sum of the five values to its right; in particular, column ASSOC corresponds to user category, while column TRES corresponds to the number of CPUs. Note that the latter contribution to priority is &amp;#039;&amp;#039;negative&amp;#039;&amp;#039;, i.e. it is a penalty.&lt;br /&gt;
&lt;br /&gt;
=== How FairShare works ===&lt;br /&gt;
In Mufasa 2.0, the FairShare of each user is computed according to these rules:&lt;br /&gt;
# It considers the quantity used by (current and past) user jobs of:&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;CPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of CPUs used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;RAM&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the amount of RAM used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;GPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of GPUs used&lt;br /&gt;
#** note: 40 GB GPUs have double the impact on FairShare than 20 GB GPUs&lt;br /&gt;
# The impact on FairShare of any use of resources is &amp;#039;&amp;#039;&amp;#039;proportional to its duration&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
#* example: using 32 GB of RAM for 48h has the same impact of using 64 GB of RAM for 24h&lt;br /&gt;
# FairShare has a &amp;quot;fading memory&amp;quot;: i.e., resource use has less and less impact on FairShare the farther in the past it is&lt;br /&gt;
#* consequence: over time, the &amp;quot;history&amp;quot; of a user gets forgotten by FairShare&lt;br /&gt;
&lt;br /&gt;
== How to maximise the priority of your jobs ==&lt;br /&gt;
&lt;br /&gt;
Every time you run a SLURM job, follow these guidelines:&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Choose the less powerful QOS compatible with the needs of your job&lt;br /&gt;
:: QOSes with access to less resources have higher base priority&lt;br /&gt;
&lt;br /&gt;
; Only request CPUs and RAM that your job really needs&lt;br /&gt;
:: Usually, code exploits multiple CPUs only if specifically designed to do so: if unsure, only ask for 1 CPU&lt;br /&gt;
:: The fewer resources your job asks for, the less it will wait before they become available&lt;br /&gt;
:: The more CPUs your job asks for, the stronger the penalty to its priority&lt;br /&gt;
:: Asking for fewer CPUs and/or less RAM improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Do not request more time than your jobs needs to complete&lt;br /&gt;
:: Make a worst-case estimate and only ask for that duration&lt;br /&gt;
:: Asking for less of Mufasa&amp;#039;s time improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Debug and test code using less powerful QOSes before running it with more powerful QOSes&lt;br /&gt;
:: Your test jobs will get a higher priority and your FairShare will improve&lt;br /&gt;
&lt;br /&gt;
; Cancel jobs that you don&amp;#039;t need them anymore&lt;br /&gt;
:: Always use [[User_Jobs#Cancelling_a_job_with_scancel|scancel]] to delete completed (or crashed) jobs: your Fairshare will improve&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Protip: it&amp;#039;s a good idea to [[User Jobs#Looking for unused GPUs|check for unused GPUs]] before choosing what to request. Requesting a GPU that is currently idle will help your job get executed sooner.&lt;br /&gt;
&lt;br /&gt;
= SLURM partitions =&lt;br /&gt;
&lt;br /&gt;
Partitions are another mechanism provided by SLURM to create different levels of access to system resources. Since in Mufasa 2.0 access to resources is controlled via [[#SLURM Quality of Service (QOS)|QOSes]], partitions are not very relevant. (This is one of the main differences between Mufasa 1.0 and Mufasa 2.0.)&lt;br /&gt;
&lt;br /&gt;
Note, however, that the default values for some features of SLURM jobs (e.g., duration) are [[#Default values|set by the partition]].&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, there is a single SLURM partition, called &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt;, and all jobs run on it. The state of &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt; can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sinfo -o &amp;quot;%10P %5a %9T %11L %10l&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PARTITION  AVAIL STATE     DEFAULTTIME TIMELIMIT &lt;br /&gt;
jobs*      up    idle      1:00:00     3-00:00:00&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where columns correspond to the following information:&lt;br /&gt;
&lt;br /&gt;
:; PARTITION&lt;br /&gt;
:: name of the partition; the asterisks indicates that it&amp;#039;s the default one&lt;br /&gt;
&lt;br /&gt;
:; AVAIL&lt;br /&gt;
:: state/availability of the partition: see [[#Partition availability|below]]&lt;br /&gt;
&lt;br /&gt;
:; STATE&lt;br /&gt;
:: state (using [https://slurm.schedmd.com/sinfo.html#SECTION_NODE-STATE-CODES these codes])&lt;br /&gt;
:: typical values are &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;mixed&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that some of the resources are busy executing jobs while other are idle, and &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;allocated&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that all of the resources are in use&lt;br /&gt;
&lt;br /&gt;
:; DEFAULTTIME&lt;br /&gt;
:: default runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:; TIMELIMIT&lt;br /&gt;
:: maximum runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
The asterisk at the end of the partition name indicates the default partition, i.e. the one on which jobs which do not ask for a specific partition are run.&lt;br /&gt;
&lt;br /&gt;
Command &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; does not tell you about the &amp;#039;&amp;#039;jobs&amp;#039;&amp;#039; submitted to a partition. This information is obtained, instead, with [[User Jobs#Inspecting jobs with squeue|command &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;]].&lt;br /&gt;
&lt;br /&gt;
== Partition availability ==&lt;br /&gt;
&lt;br /&gt;
The most important information that &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; provides is the &amp;#039;&amp;#039;&amp;#039;availability&amp;#039;&amp;#039;&amp;#039; (also called &amp;#039;&amp;#039;&amp;#039;state&amp;#039;&amp;#039;&amp;#039;) of partitions. This is shown in column &amp;quot;AVAIL&amp;quot;. Possible partition states are:&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is available&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed as soon as resources allow&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is in the process of becoming unavailable (i.e., of entering the &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; state: see below)&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is unavailable&lt;br /&gt;
:: There are no running jobs&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
When a partition goes from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; no harm is done to running jobs. In a normally functioning SLURM system, the passage from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; happens only when no jobs are running on the partition. If (e.g., due to a malfunction) the passage happens with jobs still running, they get killed.&lt;br /&gt;
&lt;br /&gt;
A partition in state &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; requires intervention by a [[Roles|Job Administrator]] to be restored to &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
== Default values ==&lt;br /&gt;
&lt;br /&gt;
The features of SLURM partitions, including the &amp;#039;&amp;#039;&amp;#039;default values&amp;#039;&amp;#039;&amp;#039; which are applied to jobs that do not make explicit requests, can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scontrol show partition&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PartitionName=jobs&lt;br /&gt;
   AllowGroups=ALL AllowAccounts=ALL AllowQos=nogpu,gpulight,gpu,gpuwide,gpuheavy-20,gpuheavy-40&lt;br /&gt;
   AllocNodes=ALL Default=YES QoS=N/A&lt;br /&gt;
   DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO&lt;br /&gt;
   MaxNodes=UNLIMITED MaxTime=3-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED&lt;br /&gt;
   Nodes=gn01&lt;br /&gt;
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO&lt;br /&gt;
   OverTimeLimit=NONE PreemptMode=OFF&lt;br /&gt;
   State=UP TotalCPUs=48 TotalNodes=1 SelectTypeParameters=NONE&lt;br /&gt;
   JobDefaults=(null)&lt;br /&gt;
   DefMemPerNode=4096 MaxMemPerNode=UNLIMITED&lt;br /&gt;
   TRES=cpu=48,mem=1011435M,node=1,billing=49,gres/gpu=13,gres/gpu:3g.20gb=5,gres/gpu:40gb=3,gres/gpu:4g.20gb=5&lt;br /&gt;
   TRESBillingWeights=cpu=1.0,gres/gpu:3g.20gb=6.0,gres/gpu:4g.20gb=6.0,gres/gpu:40gb=6.0,mem=0.05g&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In the output of &amp;lt;code&amp;gt;scontrol show partition&amp;lt;/code&amp;gt; the most relevant default values for Mufasa users are:&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefaultTime&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default execution time assigned to a job run on the partition (e.g., 1 hour)&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefMemPerNode&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default amount of RAM assigned to a job run on the partition (e.g., 4GB)&lt;br /&gt;
&lt;br /&gt;
= System resources subjected to limitations =&lt;br /&gt;
&lt;br /&gt;
In systems based on SLURM like Mufasa, &amp;#039;&amp;#039;&amp;#039;TRES (Trackable RESources)&amp;#039;&amp;#039;&amp;#039; are (from [https://slurm.schedmd.com/tres.html SLURM&amp;#039;s documentation] &amp;quot;&amp;#039;&amp;#039;resources that can be tracked for usage or used to enforce limits against.&amp;#039;&amp;#039;&amp;quot;&lt;br /&gt;
&lt;br /&gt;
TRES include CPUs, RAM and &amp;#039;&amp;#039;&amp;#039;GRES&amp;#039;&amp;#039;&amp;#039;. The last term stands for &amp;#039;&amp;#039;Generic RESources&amp;#039;&amp;#039; that a job may need for its execution. In Mufasa, the only &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; resources are the GPUs.&lt;br /&gt;
&lt;br /&gt;
== &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax ==&lt;br /&gt;
&lt;br /&gt;
To ask SLURM to assign GRES resources (i.e., GPUs) to a job, a special syntax must be used. Precisely, the name of each GPU resource takes the form&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:Name:Type&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
Considering the [[System#CPUs and GPUs|GPU complement of Mufasa]], &amp;lt;code&amp;gt;Type&amp;lt;/code&amp;gt; takes the following values:&lt;br /&gt;
&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:40gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 40 Gbytes of RAM&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:4g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 4 compute units&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 3 compute units&lt;br /&gt;
&lt;br /&gt;
So, for instance,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
identifies a resource corresponding to a GPU with 20 GB of RAM and 3 compute units.&lt;br /&gt;
&lt;br /&gt;
When asking for a GRES resource (e.g., in an &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command or an &amp;lt;code&amp;gt;SBATCH&amp;lt;/code&amp;gt; directive of an [[User Jobs#Using execution scripts to run jobs|execution script]]), the syntax required by SLURM is&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:&amp;lt;Type&amp;gt;:&amp;lt;Quantity&amp;gt;&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
where &amp;lt;code&amp;gt;Quantity&amp;lt;/code&amp;gt; is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of type &amp;lt;code&amp;gt;4g.20gb&amp;lt;/code&amp;gt; the syntax is&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:4g.20gb:2&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SLURM&amp;#039;s &amp;#039;&amp;#039;generic resources&amp;#039;&amp;#039; are defined in &amp;lt;code&amp;gt;/etc/slurm/gres.conf&amp;lt;/code&amp;gt;. In order to make GPUs available to SLURM&amp;#039;s &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; management, Mufasa makes use of Nvidia&amp;#039;s [https://developer.nvidia.com/nvidia-management-library-nvml NVML library]. For additional information see [https://slurm.schedmd.com/gres.html SLURM&amp;#039;s documentation].&lt;/div&gt;</summary>
		<author><name>GiulioFontana</name></author>
	</entry>
	<entry>
		<id>https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2586</id>
		<title>SLURM</title>
		<link rel="alternate" type="text/html" href="https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2586"/>
		<updated>2026-11-03T15:49:16Z</updated>

		<summary type="html">&lt;p&gt;GiulioFontana: /* Job priority */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page presents the features of SLURM that are most relevant to Mufasa&amp;#039;s [[Roles|Job Users]]. Job Users can submit jobs for execution, cancel their own jobs, and see other users&amp;#039; jobs (but not intervene on them).&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa &amp;#039;&amp;#039;&amp;#039;must use SLURM&amp;#039;&amp;#039;&amp;#039; to run resource-heavy processes, i.e. computing jobs that require one or more of the following:&lt;br /&gt;
* GPUs&lt;br /&gt;
* multiple CPUs&lt;br /&gt;
* powerful CPUs&lt;br /&gt;
* a significant amount of RAM&lt;br /&gt;
&lt;br /&gt;
In fact, only processes run via SLURM have access to all the resources of Mufasa. Processes run outside SLURM are executed by the [[System#Login server|login server]] virtual machine, which has minimal resources and no GPUs. Using SLURM is therefore the only way to execute resource-heavy jobs on Mufasa (this is a key difference between Mufasa 1.0 and Mufasa 2.0).&lt;br /&gt;
&lt;br /&gt;
= SLURM in a nutshell =&lt;br /&gt;
&lt;br /&gt;
Computation jobs on Mufasa needs to be launched via [[System#The SLURM job scheduling system|SLURM]]. SLURM provides jobs with access to the [[#System resources subjected to limitations|physical resources]] of Mufasa, such as CPUs, GPUs and RAM. Thanks to SLURM, processing jobs share system resources, optimising their occupation and availability. &lt;br /&gt;
&lt;br /&gt;
When a user runs a job, the job does not get executed immediately and is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039;. SLURM executes jobs according to their order in the queue: the top job in the queue gets executed as soon as the necessary resources are available, while jobs lower in the queue wait longer. The position of a job in the queue is due to the &amp;#039;&amp;#039;&amp;#039;[[#Job priority|priority]]&amp;#039;&amp;#039;&amp;#039; assigned to it by SLURM, with higher-priority jobs closer to the top. As a general rule:&lt;br /&gt;
&lt;br /&gt;
;: &amp;#039;&amp;#039;&amp;#039;the greater the fraction of Mufasa&amp;#039;s overall resources that a job asks for, the lower the job&amp;#039;s priority will be&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
The priority mechanism is used to encourage users to use Mufasa&amp;#039;s resources (i.e.: GPUs, CPUs, RAM, execution time) in an effective and equitable manner. This page includes a [[#How_to_maximise_the_priority_of_your_jobs|chart explaining how to maximise the priority of your jobs]].&lt;br /&gt;
&lt;br /&gt;
The &amp;#039;&amp;#039;&amp;#039;time&amp;#039;&amp;#039;&amp;#039; available to a job for its execution is controlled by SLURM. When a user requests execution of a job, they must specify the duration of the time slot that the job needs. The job must complete its execution before the end of the requested time slot, because at the end of such slot it gets killed by SLURM.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0 access to system resources is managed via SLURM&amp;#039;s [[#SLURM Quality of Service (QOS)|Quality of Service (QOS)]] mechanism (Mufasa 1.0 used [[#SLURM_partitions|partitions]] instead). While [[User Jobs#Running jobs with SLURM|launching a processing job via SLURM]], the user must always specify the chosen QOS. QOSes differ in the set of resources that they provide access to because each of them is designed to fit a given type of job.&lt;br /&gt;
&lt;br /&gt;
Mufasa sets limits to the number of jobs by the same user. This page includes a [[#Limits on jobs by the same user|table summarising such limits]].&lt;br /&gt;
&lt;br /&gt;
= SLURM Quality of Service (QOS) =&lt;br /&gt;
&lt;br /&gt;
Through &amp;#039;&amp;#039;&amp;#039;Quality of Services&amp;#039;&amp;#039;&amp;#039; (&amp;#039;&amp;#039;&amp;#039;QOSes&amp;#039;&amp;#039;&amp;#039;), SLURM lets system configurators assign a name to a set of related constraints.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, QOSes are used to define different levels of access to the server&amp;#039;s resources. When [[User Jobs|executing a job with SLURM]], a user must always &amp;#039;&amp;#039;&amp;#039;specify the QOS&amp;#039;&amp;#039;&amp;#039; that their job will use: this choice, in turn, determines what resources the job is able to access and influences the [[#Job priority|priority]] of the job.&lt;br /&gt;
&lt;br /&gt;
Mufasa&amp;#039;s QOSes and their features can be inspected with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list qos format=name%-11,priority,MaxSubmit,maxwall,maxtres%-80,mintres%-18&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following (see below for the same data in table form):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
Name          Priority MaxSubmit     MaxWall MaxTRES                                                                          MinTRES            &lt;br /&gt;
----------- ---------- --------- ----------- -------------------------------------------------------------------------------- ------------------ &lt;br /&gt;
normal               0                                                                                                                           &lt;br /&gt;
nogpu                4         1  3-00:00:00 cpu=16,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=128G                    &lt;br /&gt;
gpuheavy-20          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=2,mem=128G             gres/gpu:4g.20gb=1 &lt;br /&gt;
gpuheavy-40          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=1,gres/gpu:4g.20gb=0,mem=128G             gres/gpu:40gb=1    &lt;br /&gt;
gpulight             8         1    12:00:00 cpu=2,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpu                  2         1  1-00:00:00 cpu=8,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpuwide              2         2  1-00:00:00 cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=1,mem=64G              gres/gpu:4g.20gb=1 &lt;br /&gt;
build               32         1    02:00:00 cpu=2,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=16G&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The columns of this output are the following:&lt;br /&gt;
&lt;br /&gt;
:; Name&lt;br /&gt;
:: name of the QOS&lt;br /&gt;
&lt;br /&gt;
:; Priority&lt;br /&gt;
:: priority tier associated to the QOS (higher value = higher priority): see [[#Job priority|Job priority]] for details&lt;br /&gt;
&lt;br /&gt;
:; MaxSubmit&lt;br /&gt;
:: maximum number of jobs from a single user that can be submitted to SLURM with this QOS; submitted jobs include both running and queued jobs&lt;br /&gt;
:: See [[#Limits on jobs by the same user|Limits on jobs by the same user]] for an overview of the limits on jobs set by Mufasa.&lt;br /&gt;
&lt;br /&gt;
:; MaxWall&lt;br /&gt;
:: maximum wall clock duration of the jobs using the QOS (after which they are killed by SLURM), in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
:: For some QOSes these are not set: it means that they are determined by the [[#SLURM partitions|partition]]. Partitions also define the [[#Default values|default duration]] of jobs.&lt;br /&gt;
&lt;br /&gt;
:; MaxTRES&lt;br /&gt;
:: amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) available to a job using the QOS, where&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;cpu=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of CPUs (i.e., processor cores) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of CPUs specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;gres/&amp;#039;&amp;#039;gpu:Type&amp;#039;&amp;#039;=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of GPUs of class &amp;lt;code&amp;gt;&amp;#039;&amp;#039;Type&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; (see [[User Jobs#gres syntax|&amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax]]) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; (for QOSes that allow access to GPUs) if not specified, the job cannot be launched&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;mem=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;G&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum amount of system RAM is &amp;#039;&amp;#039;K&amp;#039;&amp;#039; GBytes&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of RAM specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
&lt;br /&gt;
:; MinTRES&lt;br /&gt;
:: minimum amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) that a job using the QOS must request in order to actually get executed by SLURM.&lt;br /&gt;
:: (If your job does not actually need these resources, you&amp;#039;ve chosen the wrong QOS: you can use one with a higher priority.)&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; QOS is the default one, and exists only to ensure that users always specify a QOS when running a job. Since &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; has zero priority and no resources, a job run using this QOS would never be run.&lt;br /&gt;
&lt;br /&gt;
The information provided by the &amp;lt;code&amp;gt;sacctmgr list qos&amp;lt;/code&amp;gt; command above is summarised by the following table:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! QOS&lt;br /&gt;
! Priority&amp;lt;/br&amp;gt;tier&lt;br /&gt;
! Max&amp;lt;/br&amp;gt;Submit&lt;br /&gt;
! MaxWall&amp;lt;/br&amp;gt;[h]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;CPUs&lt;br /&gt;
! max RAM&amp;lt;/br&amp;gt;[GB]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;3g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;4g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;40GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | build&lt;br /&gt;
| 32&lt;br /&gt;
| 1&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 16&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpulight&lt;br /&gt;
| 8&lt;br /&gt;
| 1&lt;br /&gt;
| 12&lt;br /&gt;
| 2&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | nogpu&lt;br /&gt;
| 4&lt;br /&gt;
| 1&lt;br /&gt;
| 72&lt;br /&gt;
| 16&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpu&lt;br /&gt;
| 2&lt;br /&gt;
| 1&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuwide&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-20&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| 2 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-40&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
A key piece of information in the table above is the &amp;#039;&amp;#039;&amp;#039;priority tier&amp;#039;&amp;#039;&amp;#039; associated to each QOS. In Mufasa 2.0, priority tiers are used to encourage users to use the &amp;#039;&amp;#039;&amp;#039;least powerful QOS that is compatible with their needs&amp;#039;&amp;#039;&amp;#039;, where &amp;quot;powerful&amp;quot; means &amp;quot;rich with resources&amp;quot;. Less powerful QOSes increase the priority of the jobs that use them, so these jobs tend to be executed sooner. See [[#Job priority|Job priority]] for details about how priority affects the execution order of jobs in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
== The &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS ==&lt;br /&gt;
&lt;br /&gt;
This QOS is specifically designed to be used by Mufasa users to &amp;#039;&amp;#039;&amp;#039;quickly build [[System#Containers|container images]]&amp;#039;&amp;#039;&amp;#039;. Its associated priority tier is very high: this, combined with the fact that jobs using this QOS require few resources, means that such jobs usually get executed very soon.&lt;br /&gt;
&lt;br /&gt;
On the other hand, the limited resources and the lack of access to the GPUs make the &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS unsuitable for other tasks.&lt;br /&gt;
&lt;br /&gt;
See [[Singularity#Building Singularity images|Building Singularity images]] for directions about building Singularity container images.&lt;br /&gt;
&lt;br /&gt;
== Restricted QOSes ==&lt;br /&gt;
&lt;br /&gt;
In Mufasa, the most powerful QOSes are reserved to researchers (called &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users: these include academic personnel and Ph.D. students). M.Sc. students, called &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users, cannot use them while running jobs.&lt;br /&gt;
&lt;br /&gt;
See [[#research users and students users|below]] to understand how &amp;#039;&amp;#039;user categories&amp;#039;&amp;#039; work in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
= &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users =&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa belong to two &amp;#039;&amp;#039;&amp;#039;user categories&amp;#039;&amp;#039;&amp;#039;. Categories provide users with different access to  to Mufasa&amp;#039;s resources: the idea being to provide researchers with more access while still letting students benefit from the server&amp;#039;s capabilities.&lt;br /&gt;
&lt;br /&gt;
User categories in Mufasa are the following:&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. academic personnel and Ph.D. students&lt;br /&gt;
::: * can use all QOSes, including the the [[#Restricted QOSes|restricted ones]]&lt;br /&gt;
::: * their jobs have a higher &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a higher number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. M.Sc. students&lt;br /&gt;
::: * cannot use the [[#Restricted QOSes|restricted QOSes]]&lt;br /&gt;
::: * their jobs have a lower &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a lower number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
You can inspect the differences in priority and running jobs between &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association format=&amp;quot;account,priority,maxjobs&amp;quot; | grep -E &amp;#039;Account|research|students&amp;#039;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
   Account   Priority MaxJobs &lt;br /&gt;
  research          4       2 &lt;br /&gt;
  students          1       1&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
To know what limits apply to your own user, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association where user=$USER format=&amp;quot;user,priority,maxjobs,qos%-60&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
      User   Priority MaxJobs QOS                                                          &lt;br /&gt;
---------- ---------- ------- ------------------------------------------------------------ &lt;br /&gt;
    preali          4       2 build,gpu,gpuheavy-20,gpuheavy-40,gpulight,gpuwide,nogpu&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The list under &amp;quot;QOS&amp;quot; shows what QOSes your user is allowed to use when [[User Jobs|running jobs]]. &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users can use all of them, while &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users can only access a subset of them.&lt;br /&gt;
&lt;br /&gt;
= Limits on jobs by the same user =&lt;br /&gt;
&lt;br /&gt;
Mufasa enforces limits on the number of jobs from a single user. Such limits aim at preventing users from &amp;quot;hogging&amp;quot; system resources, and apply to:&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;submitted jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that the user asked SLURM to execute, each of which may currently be either running or queued&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;running jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that are currently in execution&lt;br /&gt;
&lt;br /&gt;
Limits on submitted jobs are set via [[#SLURM Quality of Service (QOS)|QOSes]], while limits on running jobs are set via [[#research users and students users|user category]].&lt;br /&gt;
&lt;br /&gt;
The following table summarises what limits exist:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! &lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;running&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;submitted&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | global limits&amp;lt;br/&amp;gt;(system-wide)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to limits set by individual QOSes (below)&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | limits for&amp;lt;br/&amp;gt;individual&amp;lt;/br&amp;gt;QOSes&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to global limits (above)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for the&amp;#039;&amp;#039; &amp;lt;code&amp;gt;gpuwide&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;QOS&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for each of the other QOSes&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
= Job priority =&lt;br /&gt;
&lt;br /&gt;
Once the execution of a job has been requested, the job is not run immediately: it is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039; by SLURM, together with all the other jobs awaiting execution. The job on top of the queue at any time is the first to be put into execution as soon as the resources it requires are available. &lt;br /&gt;
&lt;br /&gt;
The order of the items in the job queue depends on their &amp;#039;&amp;#039;&amp;#039;priority&amp;#039;&amp;#039;&amp;#039;. The job with the highest priority is put by SLURM on top of the queue, while all other jobs are put in the queue in order of descending priority.&lt;br /&gt;
&lt;br /&gt;
The goal of SLURM is to maximise resource availability: i.e., to ensure the shorter possible wait time before job execution. To achieve this goal, in Mufasa SLURM is configured to &amp;#039;&amp;#039;&amp;#039;encourage users not to ask for resources or execution time that their job doesn&amp;#039;t need&amp;#039;&amp;#039;&amp;#039;. This is done via the priority mechanism: the more resources and/or time a job requests, the lower its priority will be; and the later it will be executed.&lt;br /&gt;
&lt;br /&gt;
Priority management in Mufasa is designed to set up a &amp;#039;&amp;#039;&amp;#039;virtuous cycle&amp;#039;&amp;#039;&amp;#039; where users, by carefully choosing what to ask for, obtain two results:&lt;br /&gt;
* they ensure that their job is executed as soon as possible;&lt;br /&gt;
* they leave as much as possible of Mufasa&amp;#039;s resources free for other users&amp;#039;s jobs.&lt;br /&gt;
&lt;br /&gt;
== Elements determining job priority ==&lt;br /&gt;
In Mufasa, the priority of a job is computed by SLURM according to the following elements:&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#research users and students users|User category]]&amp;#039;&amp;#039;&amp;#039; (i.e., &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;)&lt;br /&gt;
::: Used to provide higher priority to jobs run by &amp;#039;&amp;#039;&amp;#039;research personnel&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#SLURM Quality of Service (QOS)|QOS]]&amp;#039;&amp;#039;&amp;#039; used by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;less resources&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Number of CPUs&amp;#039;&amp;#039;&amp;#039; requested by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;a lower number of CPUs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job Age&amp;#039;&amp;#039;&amp;#039;, i.e. the time that the job has been waiting in the queue&lt;br /&gt;
::: Used to provide higher priority to jobs which have been &amp;#039;&amp;#039;&amp;#039;queued for a longer time&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;FairShare&amp;#039;&amp;#039;&amp;#039;, i.e. a factor computed by SLURM to balance use of the system by different users&lt;br /&gt;
::: Used to provide higher priority to jobs by users who &amp;#039;&amp;#039;&amp;#039;used Mufasa&amp;#039;s resources less than others&amp;#039;&amp;#039;&amp;#039; (see [[#How FairShare works|below]])&lt;br /&gt;
&lt;br /&gt;
You can understand how each of these elements contributes to the priority of the jobs waiting for execution (both yours and of other users) with command &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;. A tailored version of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; that only displays the parameters listed above is the following:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sprio -o &amp;quot;%6i %9u %.5Y %.10B %.5Q %.9T %.5A %.5F&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Important!&lt;br /&gt;
:: The output of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; is only relevant for jobs &amp;#039;&amp;#039;&amp;#039;not yet in execution&amp;#039;&amp;#039;&amp;#039;. For an already running job, the values shown by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; correspond to the priority that the job would get if computed now, which generally differs from the actual priority.&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Here is an example of the output of the command above:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
JOBID  USER      PRIOR      ASSOC   QOS      TRES   AGE FAIRS&lt;br /&gt;
5774   itombini   3492       2500   750  cpu=-800   906   136&lt;br /&gt;
5780   fbottazzi  2266        625   750  cpu=-600   420  1071&lt;br /&gt;
5783   mzoccola   1766        625   750  cpu=-800   217   974&lt;br /&gt;
5790   amoglia    2717       2500   375  cpu=-800   156   487&lt;br /&gt;
5791   qliu       2232       2500   375  cpu=-800   138    19&lt;br /&gt;
5792   fpastori   2197       2500   375  cpu=-800    84    39&lt;br /&gt;
5793   bagostini  2034        625  1500 cpu=-1200    18  1091&lt;br /&gt;
5794   lpassoni   2517       2500   750  cpu=-800     9    58&lt;br /&gt;
5795   lpassoni   2517       2500   750  cpu=-800     9    58&lt;br /&gt;
5796   adurat     3101       2500   750  cpu=-800     8   643&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
In it, column PRIOR provides the overall priority of each job. Each PRIOR value is the sum of the five values to its right; in particular, column ASSOC corresponds to user category, while column TRES corresponds to the number of CPUs. Note that the latter contribution to priority is &amp;#039;&amp;#039;negative&amp;#039;&amp;#039;, i.e. it is a penalty.&lt;br /&gt;
&lt;br /&gt;
=== How FairShare works ===&lt;br /&gt;
In Mufasa 2.0, the FairShare of each user is computed according to these rules:&lt;br /&gt;
# It considers the quantity used by (current and past) user jobs of:&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;CPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of CPUs used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;RAM&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the amount of RAM used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;GPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of GPUs used&lt;br /&gt;
#** note: 40 GB GPUs have double the impact on FairShare than 20 GB GPUs&lt;br /&gt;
# The impact on FairShare of any use of resources is &amp;#039;&amp;#039;&amp;#039;proportional to its duration&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
#* example: using 32 GB of RAM for 48h has the same impact of using 64 GB of RAM for 24h&lt;br /&gt;
# FairShare has a &amp;quot;fading memory&amp;quot;: i.e., resource use has less and less impact on FairShare the farther in the past it is&lt;br /&gt;
#* consequence: over time, the &amp;quot;history&amp;quot; of a user gets forgotten by FairShare&lt;br /&gt;
&lt;br /&gt;
== How to maximise the priority of your jobs ==&lt;br /&gt;
&lt;br /&gt;
Every time you run a SLURM job, follow these guidelines:&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Choose the less powerful QOS compatible with the needs of your job&lt;br /&gt;
:: QOSes with access to less resources have higher base priority&lt;br /&gt;
&lt;br /&gt;
; Only request CPUs and RAM that your job really needs&lt;br /&gt;
:: Usually, code exploits multiple CPUs only if specifically designed to do so: if unsure, only ask for 1 CPU&lt;br /&gt;
:: The fewer resources your job asks for, the less it will wait before they become available&lt;br /&gt;
:: The more CPUs your job asks for, the stronger the penalty to its priority&lt;br /&gt;
:: Asking for fewer CPUs and/or less RAM improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Do not request more time than your jobs needs to complete&lt;br /&gt;
:: Make a worst-case estimate and only ask for that duration&lt;br /&gt;
:: Asking for less of Mufasa&amp;#039;s time improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Debug and test code using less powerful QOSes before running it with more powerful QOSes&lt;br /&gt;
:: Your test jobs will get a higher priority and your FairShare will improve&lt;br /&gt;
&lt;br /&gt;
; Cancel jobs that you don&amp;#039;t need them anymore&lt;br /&gt;
:: Always use [[User_Jobs#Cancelling_a_job_with_scancel|scancel]] to delete completed (or crashed) jobs: your Fairshare will improve&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Protip: it&amp;#039;s a good idea to [[User Jobs#Looking for unused GPUs|check for unused GPUs]] before choosing what to request. Requesting a GPU that is currently idle will help your job get executed sooner.&lt;br /&gt;
&lt;br /&gt;
= SLURM partitions =&lt;br /&gt;
&lt;br /&gt;
Partitions are another mechanism provided by SLURM to create different levels of access to system resources. Since in Mufasa 2.0 access to resources is controlled via [[#SLURM Quality of Service (QOS)|QOSes]], partitions are not very relevant. (This is one of the main differences between Mufasa 1.0 and Mufasa 2.0.)&lt;br /&gt;
&lt;br /&gt;
Note, however, that the default values for some features of SLURM jobs (e.g., duration) are [[#Default values|set by the partition]].&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, there is a single SLURM partition, called &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt;, and all jobs run on it. The state of &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt; can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sinfo -o &amp;quot;%10P %5a %9T %11L %10l&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PARTITION  AVAIL STATE     DEFAULTTIME TIMELIMIT &lt;br /&gt;
jobs*      up    idle      1:00:00     3-00:00:00&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where columns correspond to the following information:&lt;br /&gt;
&lt;br /&gt;
:; PARTITION&lt;br /&gt;
:: name of the partition; the asterisks indicates that it&amp;#039;s the default one&lt;br /&gt;
&lt;br /&gt;
:; AVAIL&lt;br /&gt;
:: state/availability of the partition: see [[#Partition availability|below]]&lt;br /&gt;
&lt;br /&gt;
:; STATE&lt;br /&gt;
:: state (using [https://slurm.schedmd.com/sinfo.html#SECTION_NODE-STATE-CODES these codes])&lt;br /&gt;
:: typical values are &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;mixed&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that some of the resources are busy executing jobs while other are idle, and &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;allocated&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that all of the resources are in use&lt;br /&gt;
&lt;br /&gt;
:; DEFAULTTIME&lt;br /&gt;
:: default runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:; TIMELIMIT&lt;br /&gt;
:: maximum runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
The asterisk at the end of the partition name indicates the default partition, i.e. the one on which jobs which do not ask for a specific partition are run.&lt;br /&gt;
&lt;br /&gt;
Command &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; does not tell you about the &amp;#039;&amp;#039;jobs&amp;#039;&amp;#039; submitted to a partition. This information is obtained, instead, with [[User Jobs#Inspecting jobs with squeue|command &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;]].&lt;br /&gt;
&lt;br /&gt;
== Partition availability ==&lt;br /&gt;
&lt;br /&gt;
The most important information that &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; provides is the &amp;#039;&amp;#039;&amp;#039;availability&amp;#039;&amp;#039;&amp;#039; (also called &amp;#039;&amp;#039;&amp;#039;state&amp;#039;&amp;#039;&amp;#039;) of partitions. This is shown in column &amp;quot;AVAIL&amp;quot;. Possible partition states are:&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is available&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed as soon as resources allow&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is in the process of becoming unavailable (i.e., of entering the &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; state: see below)&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is unavailable&lt;br /&gt;
:: There are no running jobs&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
When a partition goes from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; no harm is done to running jobs. In a normally functioning SLURM system, the passage from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; happens only when no jobs are running on the partition. If (e.g., due to a malfunction) the passage happens with jobs still running, they get killed.&lt;br /&gt;
&lt;br /&gt;
A partition in state &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; requires intervention by a [[Roles|Job Administrator]] to be restored to &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
== Default values ==&lt;br /&gt;
&lt;br /&gt;
The features of SLURM partitions, including the &amp;#039;&amp;#039;&amp;#039;default values&amp;#039;&amp;#039;&amp;#039; which are applied to jobs that do not make explicit requests, can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scontrol show partition&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PartitionName=jobs&lt;br /&gt;
   AllowGroups=ALL AllowAccounts=ALL AllowQos=nogpu,gpulight,gpu,gpuwide,gpuheavy-20,gpuheavy-40&lt;br /&gt;
   AllocNodes=ALL Default=YES QoS=N/A&lt;br /&gt;
   DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO&lt;br /&gt;
   MaxNodes=UNLIMITED MaxTime=3-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED&lt;br /&gt;
   Nodes=gn01&lt;br /&gt;
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO&lt;br /&gt;
   OverTimeLimit=NONE PreemptMode=OFF&lt;br /&gt;
   State=UP TotalCPUs=48 TotalNodes=1 SelectTypeParameters=NONE&lt;br /&gt;
   JobDefaults=(null)&lt;br /&gt;
   DefMemPerNode=4096 MaxMemPerNode=UNLIMITED&lt;br /&gt;
   TRES=cpu=48,mem=1011435M,node=1,billing=49,gres/gpu=13,gres/gpu:3g.20gb=5,gres/gpu:40gb=3,gres/gpu:4g.20gb=5&lt;br /&gt;
   TRESBillingWeights=cpu=1.0,gres/gpu:3g.20gb=6.0,gres/gpu:4g.20gb=6.0,gres/gpu:40gb=6.0,mem=0.05g&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In the output of &amp;lt;code&amp;gt;scontrol show partition&amp;lt;/code&amp;gt; the most relevant default values for Mufasa users are:&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefaultTime&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default execution time assigned to a job run on the partition (e.g., 1 hour)&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefMemPerNode&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default amount of RAM assigned to a job run on the partition (e.g., 4GB)&lt;br /&gt;
&lt;br /&gt;
= System resources subjected to limitations =&lt;br /&gt;
&lt;br /&gt;
In systems based on SLURM like Mufasa, &amp;#039;&amp;#039;&amp;#039;TRES (Trackable RESources)&amp;#039;&amp;#039;&amp;#039; are (from [https://slurm.schedmd.com/tres.html SLURM&amp;#039;s documentation] &amp;quot;&amp;#039;&amp;#039;resources that can be tracked for usage or used to enforce limits against.&amp;#039;&amp;#039;&amp;quot;&lt;br /&gt;
&lt;br /&gt;
TRES include CPUs, RAM and &amp;#039;&amp;#039;&amp;#039;GRES&amp;#039;&amp;#039;&amp;#039;. The last term stands for &amp;#039;&amp;#039;Generic RESources&amp;#039;&amp;#039; that a job may need for its execution. In Mufasa, the only &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; resources are the GPUs.&lt;br /&gt;
&lt;br /&gt;
== &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax ==&lt;br /&gt;
&lt;br /&gt;
To ask SLURM to assign GRES resources (i.e., GPUs) to a job, a special syntax must be used. Precisely, the name of each GPU resource takes the form&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:Name:Type&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
Considering the [[System#CPUs and GPUs|GPU complement of Mufasa]], &amp;lt;code&amp;gt;Type&amp;lt;/code&amp;gt; takes the following values:&lt;br /&gt;
&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:40gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 40 Gbytes of RAM&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:4g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 4 compute units&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 3 compute units&lt;br /&gt;
&lt;br /&gt;
So, for instance,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
identifies a resource corresponding to a GPU with 20 GB of RAM and 3 compute units.&lt;br /&gt;
&lt;br /&gt;
When asking for a GRES resource (e.g., in an &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command or an &amp;lt;code&amp;gt;SBATCH&amp;lt;/code&amp;gt; directive of an [[User Jobs#Using execution scripts to run jobs|execution script]]), the syntax required by SLURM is&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:&amp;lt;Type&amp;gt;:&amp;lt;Quantity&amp;gt;&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
where &amp;lt;code&amp;gt;Quantity&amp;lt;/code&amp;gt; is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of type &amp;lt;code&amp;gt;4g.20gb&amp;lt;/code&amp;gt; the syntax is&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:4g.20gb:2&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SLURM&amp;#039;s &amp;#039;&amp;#039;generic resources&amp;#039;&amp;#039; are defined in &amp;lt;code&amp;gt;/etc/slurm/gres.conf&amp;lt;/code&amp;gt;. In order to make GPUs available to SLURM&amp;#039;s &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; management, Mufasa makes use of Nvidia&amp;#039;s [https://developer.nvidia.com/nvidia-management-library-nvml NVML library]. For additional information see [https://slurm.schedmd.com/gres.html SLURM&amp;#039;s documentation].&lt;/div&gt;</summary>
		<author><name>GiulioFontana</name></author>
	</entry>
	<entry>
		<id>https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2585</id>
		<title>SLURM</title>
		<link rel="alternate" type="text/html" href="https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2585"/>
		<updated>2026-11-03T15:48:17Z</updated>

		<summary type="html">&lt;p&gt;GiulioFontana: /* Job priority */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page presents the features of SLURM that are most relevant to Mufasa&amp;#039;s [[Roles|Job Users]]. Job Users can submit jobs for execution, cancel their own jobs, and see other users&amp;#039; jobs (but not intervene on them).&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa &amp;#039;&amp;#039;&amp;#039;must use SLURM&amp;#039;&amp;#039;&amp;#039; to run resource-heavy processes, i.e. computing jobs that require one or more of the following:&lt;br /&gt;
* GPUs&lt;br /&gt;
* multiple CPUs&lt;br /&gt;
* powerful CPUs&lt;br /&gt;
* a significant amount of RAM&lt;br /&gt;
&lt;br /&gt;
In fact, only processes run via SLURM have access to all the resources of Mufasa. Processes run outside SLURM are executed by the [[System#Login server|login server]] virtual machine, which has minimal resources and no GPUs. Using SLURM is therefore the only way to execute resource-heavy jobs on Mufasa (this is a key difference between Mufasa 1.0 and Mufasa 2.0).&lt;br /&gt;
&lt;br /&gt;
= SLURM in a nutshell =&lt;br /&gt;
&lt;br /&gt;
Computation jobs on Mufasa needs to be launched via [[System#The SLURM job scheduling system|SLURM]]. SLURM provides jobs with access to the [[#System resources subjected to limitations|physical resources]] of Mufasa, such as CPUs, GPUs and RAM. Thanks to SLURM, processing jobs share system resources, optimising their occupation and availability. &lt;br /&gt;
&lt;br /&gt;
When a user runs a job, the job does not get executed immediately and is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039;. SLURM executes jobs according to their order in the queue: the top job in the queue gets executed as soon as the necessary resources are available, while jobs lower in the queue wait longer. The position of a job in the queue is due to the &amp;#039;&amp;#039;&amp;#039;[[#Job priority|priority]]&amp;#039;&amp;#039;&amp;#039; assigned to it by SLURM, with higher-priority jobs closer to the top. As a general rule:&lt;br /&gt;
&lt;br /&gt;
;: &amp;#039;&amp;#039;&amp;#039;the greater the fraction of Mufasa&amp;#039;s overall resources that a job asks for, the lower the job&amp;#039;s priority will be&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
The priority mechanism is used to encourage users to use Mufasa&amp;#039;s resources (i.e.: GPUs, CPUs, RAM, execution time) in an effective and equitable manner. This page includes a [[#How_to_maximise_the_priority_of_your_jobs|chart explaining how to maximise the priority of your jobs]].&lt;br /&gt;
&lt;br /&gt;
The &amp;#039;&amp;#039;&amp;#039;time&amp;#039;&amp;#039;&amp;#039; available to a job for its execution is controlled by SLURM. When a user requests execution of a job, they must specify the duration of the time slot that the job needs. The job must complete its execution before the end of the requested time slot, because at the end of such slot it gets killed by SLURM.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0 access to system resources is managed via SLURM&amp;#039;s [[#SLURM Quality of Service (QOS)|Quality of Service (QOS)]] mechanism (Mufasa 1.0 used [[#SLURM_partitions|partitions]] instead). While [[User Jobs#Running jobs with SLURM|launching a processing job via SLURM]], the user must always specify the chosen QOS. QOSes differ in the set of resources that they provide access to because each of them is designed to fit a given type of job.&lt;br /&gt;
&lt;br /&gt;
Mufasa sets limits to the number of jobs by the same user. This page includes a [[#Limits on jobs by the same user|table summarising such limits]].&lt;br /&gt;
&lt;br /&gt;
= SLURM Quality of Service (QOS) =&lt;br /&gt;
&lt;br /&gt;
Through &amp;#039;&amp;#039;&amp;#039;Quality of Services&amp;#039;&amp;#039;&amp;#039; (&amp;#039;&amp;#039;&amp;#039;QOSes&amp;#039;&amp;#039;&amp;#039;), SLURM lets system configurators assign a name to a set of related constraints.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, QOSes are used to define different levels of access to the server&amp;#039;s resources. When [[User Jobs|executing a job with SLURM]], a user must always &amp;#039;&amp;#039;&amp;#039;specify the QOS&amp;#039;&amp;#039;&amp;#039; that their job will use: this choice, in turn, determines what resources the job is able to access and influences the [[#Job priority|priority]] of the job.&lt;br /&gt;
&lt;br /&gt;
Mufasa&amp;#039;s QOSes and their features can be inspected with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list qos format=name%-11,priority,MaxSubmit,maxwall,maxtres%-80,mintres%-18&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following (see below for the same data in table form):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
Name          Priority MaxSubmit     MaxWall MaxTRES                                                                          MinTRES            &lt;br /&gt;
----------- ---------- --------- ----------- -------------------------------------------------------------------------------- ------------------ &lt;br /&gt;
normal               0                                                                                                                           &lt;br /&gt;
nogpu                4         1  3-00:00:00 cpu=16,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=128G                    &lt;br /&gt;
gpuheavy-20          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=2,mem=128G             gres/gpu:4g.20gb=1 &lt;br /&gt;
gpuheavy-40          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=1,gres/gpu:4g.20gb=0,mem=128G             gres/gpu:40gb=1    &lt;br /&gt;
gpulight             8         1    12:00:00 cpu=2,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpu                  2         1  1-00:00:00 cpu=8,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpuwide              2         2  1-00:00:00 cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=1,mem=64G              gres/gpu:4g.20gb=1 &lt;br /&gt;
build               32         1    02:00:00 cpu=2,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=16G&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The columns of this output are the following:&lt;br /&gt;
&lt;br /&gt;
:; Name&lt;br /&gt;
:: name of the QOS&lt;br /&gt;
&lt;br /&gt;
:; Priority&lt;br /&gt;
:: priority tier associated to the QOS (higher value = higher priority): see [[#Job priority|Job priority]] for details&lt;br /&gt;
&lt;br /&gt;
:; MaxSubmit&lt;br /&gt;
:: maximum number of jobs from a single user that can be submitted to SLURM with this QOS; submitted jobs include both running and queued jobs&lt;br /&gt;
:: See [[#Limits on jobs by the same user|Limits on jobs by the same user]] for an overview of the limits on jobs set by Mufasa.&lt;br /&gt;
&lt;br /&gt;
:; MaxWall&lt;br /&gt;
:: maximum wall clock duration of the jobs using the QOS (after which they are killed by SLURM), in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
:: For some QOSes these are not set: it means that they are determined by the [[#SLURM partitions|partition]]. Partitions also define the [[#Default values|default duration]] of jobs.&lt;br /&gt;
&lt;br /&gt;
:; MaxTRES&lt;br /&gt;
:: amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) available to a job using the QOS, where&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;cpu=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of CPUs (i.e., processor cores) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of CPUs specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;gres/&amp;#039;&amp;#039;gpu:Type&amp;#039;&amp;#039;=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of GPUs of class &amp;lt;code&amp;gt;&amp;#039;&amp;#039;Type&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; (see [[User Jobs#gres syntax|&amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax]]) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; (for QOSes that allow access to GPUs) if not specified, the job cannot be launched&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;mem=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;G&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum amount of system RAM is &amp;#039;&amp;#039;K&amp;#039;&amp;#039; GBytes&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of RAM specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
&lt;br /&gt;
:; MinTRES&lt;br /&gt;
:: minimum amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) that a job using the QOS must request in order to actually get executed by SLURM.&lt;br /&gt;
:: (If your job does not actually need these resources, you&amp;#039;ve chosen the wrong QOS: you can use one with a higher priority.)&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; QOS is the default one, and exists only to ensure that users always specify a QOS when running a job. Since &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; has zero priority and no resources, a job run using this QOS would never be run.&lt;br /&gt;
&lt;br /&gt;
The information provided by the &amp;lt;code&amp;gt;sacctmgr list qos&amp;lt;/code&amp;gt; command above is summarised by the following table:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! QOS&lt;br /&gt;
! Priority&amp;lt;/br&amp;gt;tier&lt;br /&gt;
! Max&amp;lt;/br&amp;gt;Submit&lt;br /&gt;
! MaxWall&amp;lt;/br&amp;gt;[h]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;CPUs&lt;br /&gt;
! max RAM&amp;lt;/br&amp;gt;[GB]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;3g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;4g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;40GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | build&lt;br /&gt;
| 32&lt;br /&gt;
| 1&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 16&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpulight&lt;br /&gt;
| 8&lt;br /&gt;
| 1&lt;br /&gt;
| 12&lt;br /&gt;
| 2&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | nogpu&lt;br /&gt;
| 4&lt;br /&gt;
| 1&lt;br /&gt;
| 72&lt;br /&gt;
| 16&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpu&lt;br /&gt;
| 2&lt;br /&gt;
| 1&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuwide&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-20&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| 2 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-40&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
A key piece of information in the table above is the &amp;#039;&amp;#039;&amp;#039;priority tier&amp;#039;&amp;#039;&amp;#039; associated to each QOS. In Mufasa 2.0, priority tiers are used to encourage users to use the &amp;#039;&amp;#039;&amp;#039;least powerful QOS that is compatible with their needs&amp;#039;&amp;#039;&amp;#039;, where &amp;quot;powerful&amp;quot; means &amp;quot;rich with resources&amp;quot;. Less powerful QOSes increase the priority of the jobs that use them, so these jobs tend to be executed sooner. See [[#Job priority|Job priority]] for details about how priority affects the execution order of jobs in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
== The &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS ==&lt;br /&gt;
&lt;br /&gt;
This QOS is specifically designed to be used by Mufasa users to &amp;#039;&amp;#039;&amp;#039;quickly build [[System#Containers|container images]]&amp;#039;&amp;#039;&amp;#039;. Its associated priority tier is very high: this, combined with the fact that jobs using this QOS require few resources, means that such jobs usually get executed very soon.&lt;br /&gt;
&lt;br /&gt;
On the other hand, the limited resources and the lack of access to the GPUs make the &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS unsuitable for other tasks.&lt;br /&gt;
&lt;br /&gt;
See [[Singularity#Building Singularity images|Building Singularity images]] for directions about building Singularity container images.&lt;br /&gt;
&lt;br /&gt;
== Restricted QOSes ==&lt;br /&gt;
&lt;br /&gt;
In Mufasa, the most powerful QOSes are reserved to researchers (called &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users: these include academic personnel and Ph.D. students). M.Sc. students, called &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users, cannot use them while running jobs.&lt;br /&gt;
&lt;br /&gt;
See [[#research users and students users|below]] to understand how &amp;#039;&amp;#039;user categories&amp;#039;&amp;#039; work in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
= &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users =&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa belong to two &amp;#039;&amp;#039;&amp;#039;user categories&amp;#039;&amp;#039;&amp;#039;. Categories provide users with different access to  to Mufasa&amp;#039;s resources: the idea being to provide researchers with more access while still letting students benefit from the server&amp;#039;s capabilities.&lt;br /&gt;
&lt;br /&gt;
User categories in Mufasa are the following:&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. academic personnel and Ph.D. students&lt;br /&gt;
::: * can use all QOSes, including the the [[#Restricted QOSes|restricted ones]]&lt;br /&gt;
::: * their jobs have a higher &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a higher number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. M.Sc. students&lt;br /&gt;
::: * cannot use the [[#Restricted QOSes|restricted QOSes]]&lt;br /&gt;
::: * their jobs have a lower &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a lower number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
You can inspect the differences in priority and running jobs between &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association format=&amp;quot;account,priority,maxjobs&amp;quot; | grep -E &amp;#039;Account|research|students&amp;#039;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
   Account   Priority MaxJobs &lt;br /&gt;
  research          4       2 &lt;br /&gt;
  students          1       1&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
To know what limits apply to your own user, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association where user=$USER format=&amp;quot;user,priority,maxjobs,qos%-60&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
      User   Priority MaxJobs QOS                                                          &lt;br /&gt;
---------- ---------- ------- ------------------------------------------------------------ &lt;br /&gt;
    preali          4       2 build,gpu,gpuheavy-20,gpuheavy-40,gpulight,gpuwide,nogpu&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The list under &amp;quot;QOS&amp;quot; shows what QOSes your user is allowed to use when [[User Jobs|running jobs]]. &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users can use all of them, while &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users can only access a subset of them.&lt;br /&gt;
&lt;br /&gt;
= Limits on jobs by the same user =&lt;br /&gt;
&lt;br /&gt;
Mufasa enforces limits on the number of jobs from a single user. Such limits aim at preventing users from &amp;quot;hogging&amp;quot; system resources, and apply to:&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;submitted jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that the user asked SLURM to execute, each of which may currently be either running or queued&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;running jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that are currently in execution&lt;br /&gt;
&lt;br /&gt;
Limits on submitted jobs are set via [[#SLURM Quality of Service (QOS)|QOSes]], while limits on running jobs are set via [[#research users and students users|user category]].&lt;br /&gt;
&lt;br /&gt;
The following table summarises what limits exist:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! &lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;running&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;submitted&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | global limits&amp;lt;br/&amp;gt;(system-wide)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to limits set by individual QOSes (below)&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | limits for&amp;lt;br/&amp;gt;individual&amp;lt;/br&amp;gt;QOSes&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to global limits (above)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for the&amp;#039;&amp;#039; &amp;lt;code&amp;gt;gpuwide&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;QOS&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for each of the other QOSes&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
= Job priority =&lt;br /&gt;
&lt;br /&gt;
Once the execution of a job has been requested, the job is not run immediately: it is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039; by SLURM, together with all the other jobs awaiting execution. The job on top of the queue at any time is the first to be put into execution as soon as the resources it requires are available. &lt;br /&gt;
&lt;br /&gt;
The order of the items in the job queue depends on their &amp;#039;&amp;#039;&amp;#039;priority&amp;#039;&amp;#039;&amp;#039;. The job with the highest priority is put by SLURM on top of the queue, while all other jobs are put in the queue in order of descending priority.&lt;br /&gt;
&lt;br /&gt;
The goal of SLURM is to maximise resource availability: i.e., to ensure the shorter possible wait time before job execution. To achieve this goal, in Mufasa SLURM is configured to &amp;#039;&amp;#039;&amp;#039;encourage users not to ask for resources or execution time that their job doesn&amp;#039;t need&amp;#039;&amp;#039;&amp;#039;. This is done via the priority mechanism: the more resources and/or time a job requests, the lower its priority will be; and the later it will be executed.&lt;br /&gt;
&lt;br /&gt;
Priority management in Mufasa is designed to set up a &amp;#039;&amp;#039;&amp;#039;virtuous cycle&amp;#039;&amp;#039;&amp;#039; where users, by carefully choosing what to ask for, obtain two results:&lt;br /&gt;
* they ensure that their job is executed as soon as possible;&lt;br /&gt;
* they leave as much as possible of Mufasa&amp;#039;s resources free for other users&amp;#039;s jobs.&lt;br /&gt;
&lt;br /&gt;
== Elements determining job priority ==&lt;br /&gt;
In Mufasa, the priority of a job is computed by SLURM according to the following elements:&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#research users and students users|User category]]&amp;#039;&amp;#039;&amp;#039; (i.e., &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;)&lt;br /&gt;
::: Used to provide higher priority to jobs run by &amp;#039;&amp;#039;&amp;#039;research personnel&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#SLURM Quality of Service (QOS)|QOS]]&amp;#039;&amp;#039;&amp;#039; used by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;less resources&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Number of CPUs&amp;#039;&amp;#039;&amp;#039; requested by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;a lower number of CPUs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job Age&amp;#039;&amp;#039;&amp;#039;, i.e. the time that the job has been waiting in the queue&lt;br /&gt;
::: Used to provide higher priority to jobs which have been &amp;#039;&amp;#039;&amp;#039;queued for a longer time&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;FairShare&amp;#039;&amp;#039;&amp;#039;, i.e. a factor computed by SLURM to balance use of the system by different users&lt;br /&gt;
::: Used to provide higher priority to jobs by users who &amp;#039;&amp;#039;&amp;#039;used Mufasa&amp;#039;s resources less than others&amp;#039;&amp;#039;&amp;#039; (see [[#How FairShare works|below]])&lt;br /&gt;
&lt;br /&gt;
You can understand how each of these elements contributes to the priority of the jobs waiting for execution (both yours and of other users) with command &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;. A tailored version of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; that only displays the parameters listed above is the following:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sprio -o &amp;quot;%6i %9u %.5Y %.10B %.5Q %.9T %.5A %.5F&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Important!&lt;br /&gt;
:: The output of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; is only relevant for jobs &amp;#039;&amp;#039;&amp;#039;not yet in execution&amp;#039;&amp;#039;&amp;#039;. For an already running job, the values shown by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; correspond to the priority that the job would get if computed now, which generally differs from the actual priority.&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Here is an example of the output of the command above:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
JOBID  USER      PRIOR      ASSOC   QOS      TRES   AGE FAIRS&lt;br /&gt;
5774   itombini   3492       2500   750  cpu=-800   906   136&lt;br /&gt;
5780   fbottazzi  2266        625   750  cpu=-600   420  1071&lt;br /&gt;
5783   mzoccola   1766        625   750  cpu=-800   217   974&lt;br /&gt;
5790   amoglia    2717       2500   375  cpu=-800   156   487&lt;br /&gt;
5791   qliu       2232       2500   375  cpu=-800   138    19&lt;br /&gt;
5792   fpastori   2197       2500   375  cpu=-800    84    39&lt;br /&gt;
5793   bagostini  2034        625  1500 cpu=-1200    18  1091&lt;br /&gt;
5794   lpassoni   2517       2500   750  cpu=-800     9    58&lt;br /&gt;
5795   lpassoni   2517       2500   750  cpu=-800     9    58&lt;br /&gt;
5796   adurat     3101       2500   750  cpu=-800     8   643&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
In it, column PRIOR provides the overall priority of each job. The value under PRIOR is the sum of the five values to its right; in particular, column ASSOC corresponds to user category, while column TRES corresponds to the number of CPUs. Note that the latter contribution to priority is &amp;#039;&amp;#039;negative&amp;#039;&amp;#039;, i.e. it is a penalty.&lt;br /&gt;
&lt;br /&gt;
=== How FairShare works ===&lt;br /&gt;
In Mufasa 2.0, the FairShare of each user is computed according to these rules:&lt;br /&gt;
# It considers the quantity used by (current and past) user jobs of:&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;CPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of CPUs used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;RAM&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the amount of RAM used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;GPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of GPUs used&lt;br /&gt;
#** note: 40 GB GPUs have double the impact on FairShare than 20 GB GPUs&lt;br /&gt;
# The impact on FairShare of any use of resources is &amp;#039;&amp;#039;&amp;#039;proportional to its duration&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
#* example: using 32 GB of RAM for 48h has the same impact of using 64 GB of RAM for 24h&lt;br /&gt;
# FairShare has a &amp;quot;fading memory&amp;quot;: i.e., resource use has less and less impact on FairShare the farther in the past it is&lt;br /&gt;
#* consequence: over time, the &amp;quot;history&amp;quot; of a user gets forgotten by FairShare&lt;br /&gt;
&lt;br /&gt;
== How to maximise the priority of your jobs ==&lt;br /&gt;
&lt;br /&gt;
Every time you run a SLURM job, follow these guidelines:&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Choose the less powerful QOS compatible with the needs of your job&lt;br /&gt;
:: QOSes with access to less resources have higher base priority&lt;br /&gt;
&lt;br /&gt;
; Only request CPUs and RAM that your job really needs&lt;br /&gt;
:: Usually, code exploits multiple CPUs only if specifically designed to do so: if unsure, only ask for 1 CPU&lt;br /&gt;
:: The fewer resources your job asks for, the less it will wait before they become available&lt;br /&gt;
:: The more CPUs your job asks for, the stronger the penalty to its priority&lt;br /&gt;
:: Asking for fewer CPUs and/or less RAM improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Do not request more time than your jobs needs to complete&lt;br /&gt;
:: Make a worst-case estimate and only ask for that duration&lt;br /&gt;
:: Asking for less of Mufasa&amp;#039;s time improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Debug and test code using less powerful QOSes before running it with more powerful QOSes&lt;br /&gt;
:: Your test jobs will get a higher priority and your FairShare will improve&lt;br /&gt;
&lt;br /&gt;
; Cancel jobs that you don&amp;#039;t need them anymore&lt;br /&gt;
:: Always use [[User_Jobs#Cancelling_a_job_with_scancel|scancel]] to delete completed (or crashed) jobs: your Fairshare will improve&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Protip: it&amp;#039;s a good idea to [[User Jobs#Looking for unused GPUs|check for unused GPUs]] before choosing what to request. Requesting a GPU that is currently idle will help your job get executed sooner.&lt;br /&gt;
&lt;br /&gt;
= SLURM partitions =&lt;br /&gt;
&lt;br /&gt;
Partitions are another mechanism provided by SLURM to create different levels of access to system resources. Since in Mufasa 2.0 access to resources is controlled via [[#SLURM Quality of Service (QOS)|QOSes]], partitions are not very relevant. (This is one of the main differences between Mufasa 1.0 and Mufasa 2.0.)&lt;br /&gt;
&lt;br /&gt;
Note, however, that the default values for some features of SLURM jobs (e.g., duration) are [[#Default values|set by the partition]].&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, there is a single SLURM partition, called &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt;, and all jobs run on it. The state of &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt; can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sinfo -o &amp;quot;%10P %5a %9T %11L %10l&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PARTITION  AVAIL STATE     DEFAULTTIME TIMELIMIT &lt;br /&gt;
jobs*      up    idle      1:00:00     3-00:00:00&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where columns correspond to the following information:&lt;br /&gt;
&lt;br /&gt;
:; PARTITION&lt;br /&gt;
:: name of the partition; the asterisks indicates that it&amp;#039;s the default one&lt;br /&gt;
&lt;br /&gt;
:; AVAIL&lt;br /&gt;
:: state/availability of the partition: see [[#Partition availability|below]]&lt;br /&gt;
&lt;br /&gt;
:; STATE&lt;br /&gt;
:: state (using [https://slurm.schedmd.com/sinfo.html#SECTION_NODE-STATE-CODES these codes])&lt;br /&gt;
:: typical values are &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;mixed&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that some of the resources are busy executing jobs while other are idle, and &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;allocated&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that all of the resources are in use&lt;br /&gt;
&lt;br /&gt;
:; DEFAULTTIME&lt;br /&gt;
:: default runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:; TIMELIMIT&lt;br /&gt;
:: maximum runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
The asterisk at the end of the partition name indicates the default partition, i.e. the one on which jobs which do not ask for a specific partition are run.&lt;br /&gt;
&lt;br /&gt;
Command &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; does not tell you about the &amp;#039;&amp;#039;jobs&amp;#039;&amp;#039; submitted to a partition. This information is obtained, instead, with [[User Jobs#Inspecting jobs with squeue|command &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;]].&lt;br /&gt;
&lt;br /&gt;
== Partition availability ==&lt;br /&gt;
&lt;br /&gt;
The most important information that &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; provides is the &amp;#039;&amp;#039;&amp;#039;availability&amp;#039;&amp;#039;&amp;#039; (also called &amp;#039;&amp;#039;&amp;#039;state&amp;#039;&amp;#039;&amp;#039;) of partitions. This is shown in column &amp;quot;AVAIL&amp;quot;. Possible partition states are:&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is available&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed as soon as resources allow&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is in the process of becoming unavailable (i.e., of entering the &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; state: see below)&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is unavailable&lt;br /&gt;
:: There are no running jobs&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
When a partition goes from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; no harm is done to running jobs. In a normally functioning SLURM system, the passage from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; happens only when no jobs are running on the partition. If (e.g., due to a malfunction) the passage happens with jobs still running, they get killed.&lt;br /&gt;
&lt;br /&gt;
A partition in state &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; requires intervention by a [[Roles|Job Administrator]] to be restored to &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
== Default values ==&lt;br /&gt;
&lt;br /&gt;
The features of SLURM partitions, including the &amp;#039;&amp;#039;&amp;#039;default values&amp;#039;&amp;#039;&amp;#039; which are applied to jobs that do not make explicit requests, can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scontrol show partition&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PartitionName=jobs&lt;br /&gt;
   AllowGroups=ALL AllowAccounts=ALL AllowQos=nogpu,gpulight,gpu,gpuwide,gpuheavy-20,gpuheavy-40&lt;br /&gt;
   AllocNodes=ALL Default=YES QoS=N/A&lt;br /&gt;
   DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO&lt;br /&gt;
   MaxNodes=UNLIMITED MaxTime=3-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED&lt;br /&gt;
   Nodes=gn01&lt;br /&gt;
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO&lt;br /&gt;
   OverTimeLimit=NONE PreemptMode=OFF&lt;br /&gt;
   State=UP TotalCPUs=48 TotalNodes=1 SelectTypeParameters=NONE&lt;br /&gt;
   JobDefaults=(null)&lt;br /&gt;
   DefMemPerNode=4096 MaxMemPerNode=UNLIMITED&lt;br /&gt;
   TRES=cpu=48,mem=1011435M,node=1,billing=49,gres/gpu=13,gres/gpu:3g.20gb=5,gres/gpu:40gb=3,gres/gpu:4g.20gb=5&lt;br /&gt;
   TRESBillingWeights=cpu=1.0,gres/gpu:3g.20gb=6.0,gres/gpu:4g.20gb=6.0,gres/gpu:40gb=6.0,mem=0.05g&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In the output of &amp;lt;code&amp;gt;scontrol show partition&amp;lt;/code&amp;gt; the most relevant default values for Mufasa users are:&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefaultTime&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default execution time assigned to a job run on the partition (e.g., 1 hour)&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefMemPerNode&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default amount of RAM assigned to a job run on the partition (e.g., 4GB)&lt;br /&gt;
&lt;br /&gt;
= System resources subjected to limitations =&lt;br /&gt;
&lt;br /&gt;
In systems based on SLURM like Mufasa, &amp;#039;&amp;#039;&amp;#039;TRES (Trackable RESources)&amp;#039;&amp;#039;&amp;#039; are (from [https://slurm.schedmd.com/tres.html SLURM&amp;#039;s documentation] &amp;quot;&amp;#039;&amp;#039;resources that can be tracked for usage or used to enforce limits against.&amp;#039;&amp;#039;&amp;quot;&lt;br /&gt;
&lt;br /&gt;
TRES include CPUs, RAM and &amp;#039;&amp;#039;&amp;#039;GRES&amp;#039;&amp;#039;&amp;#039;. The last term stands for &amp;#039;&amp;#039;Generic RESources&amp;#039;&amp;#039; that a job may need for its execution. In Mufasa, the only &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; resources are the GPUs.&lt;br /&gt;
&lt;br /&gt;
== &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax ==&lt;br /&gt;
&lt;br /&gt;
To ask SLURM to assign GRES resources (i.e., GPUs) to a job, a special syntax must be used. Precisely, the name of each GPU resource takes the form&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:Name:Type&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
Considering the [[System#CPUs and GPUs|GPU complement of Mufasa]], &amp;lt;code&amp;gt;Type&amp;lt;/code&amp;gt; takes the following values:&lt;br /&gt;
&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:40gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 40 Gbytes of RAM&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:4g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 4 compute units&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 3 compute units&lt;br /&gt;
&lt;br /&gt;
So, for instance,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
identifies a resource corresponding to a GPU with 20 GB of RAM and 3 compute units.&lt;br /&gt;
&lt;br /&gt;
When asking for a GRES resource (e.g., in an &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command or an &amp;lt;code&amp;gt;SBATCH&amp;lt;/code&amp;gt; directive of an [[User Jobs#Using execution scripts to run jobs|execution script]]), the syntax required by SLURM is&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:&amp;lt;Type&amp;gt;:&amp;lt;Quantity&amp;gt;&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
where &amp;lt;code&amp;gt;Quantity&amp;lt;/code&amp;gt; is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of type &amp;lt;code&amp;gt;4g.20gb&amp;lt;/code&amp;gt; the syntax is&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:4g.20gb:2&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SLURM&amp;#039;s &amp;#039;&amp;#039;generic resources&amp;#039;&amp;#039; are defined in &amp;lt;code&amp;gt;/etc/slurm/gres.conf&amp;lt;/code&amp;gt;. In order to make GPUs available to SLURM&amp;#039;s &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; management, Mufasa makes use of Nvidia&amp;#039;s [https://developer.nvidia.com/nvidia-management-library-nvml NVML library]. For additional information see [https://slurm.schedmd.com/gres.html SLURM&amp;#039;s documentation].&lt;/div&gt;</summary>
		<author><name>GiulioFontana</name></author>
	</entry>
	<entry>
		<id>https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2584</id>
		<title>SLURM</title>
		<link rel="alternate" type="text/html" href="https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2584"/>
		<updated>2026-11-03T14:53:49Z</updated>

		<summary type="html">&lt;p&gt;GiulioFontana: /* Default values */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page presents the features of SLURM that are most relevant to Mufasa&amp;#039;s [[Roles|Job Users]]. Job Users can submit jobs for execution, cancel their own jobs, and see other users&amp;#039; jobs (but not intervene on them).&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa &amp;#039;&amp;#039;&amp;#039;must use SLURM&amp;#039;&amp;#039;&amp;#039; to run resource-heavy processes, i.e. computing jobs that require one or more of the following:&lt;br /&gt;
* GPUs&lt;br /&gt;
* multiple CPUs&lt;br /&gt;
* powerful CPUs&lt;br /&gt;
* a significant amount of RAM&lt;br /&gt;
&lt;br /&gt;
In fact, only processes run via SLURM have access to all the resources of Mufasa. Processes run outside SLURM are executed by the [[System#Login server|login server]] virtual machine, which has minimal resources and no GPUs. Using SLURM is therefore the only way to execute resource-heavy jobs on Mufasa (this is a key difference between Mufasa 1.0 and Mufasa 2.0).&lt;br /&gt;
&lt;br /&gt;
= SLURM in a nutshell =&lt;br /&gt;
&lt;br /&gt;
Computation jobs on Mufasa needs to be launched via [[System#The SLURM job scheduling system|SLURM]]. SLURM provides jobs with access to the [[#System resources subjected to limitations|physical resources]] of Mufasa, such as CPUs, GPUs and RAM. Thanks to SLURM, processing jobs share system resources, optimising their occupation and availability. &lt;br /&gt;
&lt;br /&gt;
When a user runs a job, the job does not get executed immediately and is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039;. SLURM executes jobs according to their order in the queue: the top job in the queue gets executed as soon as the necessary resources are available, while jobs lower in the queue wait longer. The position of a job in the queue is due to the &amp;#039;&amp;#039;&amp;#039;[[#Job priority|priority]]&amp;#039;&amp;#039;&amp;#039; assigned to it by SLURM, with higher-priority jobs closer to the top. As a general rule:&lt;br /&gt;
&lt;br /&gt;
;: &amp;#039;&amp;#039;&amp;#039;the greater the fraction of Mufasa&amp;#039;s overall resources that a job asks for, the lower the job&amp;#039;s priority will be&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
The priority mechanism is used to encourage users to use Mufasa&amp;#039;s resources (i.e.: GPUs, CPUs, RAM, execution time) in an effective and equitable manner. This page includes a [[#How_to_maximise_the_priority_of_your_jobs|chart explaining how to maximise the priority of your jobs]].&lt;br /&gt;
&lt;br /&gt;
The &amp;#039;&amp;#039;&amp;#039;time&amp;#039;&amp;#039;&amp;#039; available to a job for its execution is controlled by SLURM. When a user requests execution of a job, they must specify the duration of the time slot that the job needs. The job must complete its execution before the end of the requested time slot, because at the end of such slot it gets killed by SLURM.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0 access to system resources is managed via SLURM&amp;#039;s [[#SLURM Quality of Service (QOS)|Quality of Service (QOS)]] mechanism (Mufasa 1.0 used [[#SLURM_partitions|partitions]] instead). While [[User Jobs#Running jobs with SLURM|launching a processing job via SLURM]], the user must always specify the chosen QOS. QOSes differ in the set of resources that they provide access to because each of them is designed to fit a given type of job.&lt;br /&gt;
&lt;br /&gt;
Mufasa sets limits to the number of jobs by the same user. This page includes a [[#Limits on jobs by the same user|table summarising such limits]].&lt;br /&gt;
&lt;br /&gt;
= SLURM Quality of Service (QOS) =&lt;br /&gt;
&lt;br /&gt;
Through &amp;#039;&amp;#039;&amp;#039;Quality of Services&amp;#039;&amp;#039;&amp;#039; (&amp;#039;&amp;#039;&amp;#039;QOSes&amp;#039;&amp;#039;&amp;#039;), SLURM lets system configurators assign a name to a set of related constraints.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, QOSes are used to define different levels of access to the server&amp;#039;s resources. When [[User Jobs|executing a job with SLURM]], a user must always &amp;#039;&amp;#039;&amp;#039;specify the QOS&amp;#039;&amp;#039;&amp;#039; that their job will use: this choice, in turn, determines what resources the job is able to access and influences the [[#Job priority|priority]] of the job.&lt;br /&gt;
&lt;br /&gt;
Mufasa&amp;#039;s QOSes and their features can be inspected with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list qos format=name%-11,priority,MaxSubmit,maxwall,maxtres%-80,mintres%-18&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following (see below for the same data in table form):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
Name          Priority MaxSubmit     MaxWall MaxTRES                                                                          MinTRES            &lt;br /&gt;
----------- ---------- --------- ----------- -------------------------------------------------------------------------------- ------------------ &lt;br /&gt;
normal               0                                                                                                                           &lt;br /&gt;
nogpu                4         1  3-00:00:00 cpu=16,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=128G                    &lt;br /&gt;
gpuheavy-20          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=2,mem=128G             gres/gpu:4g.20gb=1 &lt;br /&gt;
gpuheavy-40          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=1,gres/gpu:4g.20gb=0,mem=128G             gres/gpu:40gb=1    &lt;br /&gt;
gpulight             8         1    12:00:00 cpu=2,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpu                  2         1  1-00:00:00 cpu=8,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpuwide              2         2  1-00:00:00 cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=1,mem=64G              gres/gpu:4g.20gb=1 &lt;br /&gt;
build               32         1    02:00:00 cpu=2,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=16G&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The columns of this output are the following:&lt;br /&gt;
&lt;br /&gt;
:; Name&lt;br /&gt;
:: name of the QOS&lt;br /&gt;
&lt;br /&gt;
:; Priority&lt;br /&gt;
:: priority tier associated to the QOS (higher value = higher priority): see [[#Job priority|Job priority]] for details&lt;br /&gt;
&lt;br /&gt;
:; MaxSubmit&lt;br /&gt;
:: maximum number of jobs from a single user that can be submitted to SLURM with this QOS; submitted jobs include both running and queued jobs&lt;br /&gt;
:: See [[#Limits on jobs by the same user|Limits on jobs by the same user]] for an overview of the limits on jobs set by Mufasa.&lt;br /&gt;
&lt;br /&gt;
:; MaxWall&lt;br /&gt;
:: maximum wall clock duration of the jobs using the QOS (after which they are killed by SLURM), in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
:: For some QOSes these are not set: it means that they are determined by the [[#SLURM partitions|partition]]. Partitions also define the [[#Default values|default duration]] of jobs.&lt;br /&gt;
&lt;br /&gt;
:; MaxTRES&lt;br /&gt;
:: amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) available to a job using the QOS, where&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;cpu=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of CPUs (i.e., processor cores) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of CPUs specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;gres/&amp;#039;&amp;#039;gpu:Type&amp;#039;&amp;#039;=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of GPUs of class &amp;lt;code&amp;gt;&amp;#039;&amp;#039;Type&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; (see [[User Jobs#gres syntax|&amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax]]) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; (for QOSes that allow access to GPUs) if not specified, the job cannot be launched&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;mem=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;G&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum amount of system RAM is &amp;#039;&amp;#039;K&amp;#039;&amp;#039; GBytes&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of RAM specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
&lt;br /&gt;
:; MinTRES&lt;br /&gt;
:: minimum amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) that a job using the QOS must request in order to actually get executed by SLURM.&lt;br /&gt;
:: (If your job does not actually need these resources, you&amp;#039;ve chosen the wrong QOS: you can use one with a higher priority.)&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; QOS is the default one, and exists only to ensure that users always specify a QOS when running a job. Since &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; has zero priority and no resources, a job run using this QOS would never be run.&lt;br /&gt;
&lt;br /&gt;
The information provided by the &amp;lt;code&amp;gt;sacctmgr list qos&amp;lt;/code&amp;gt; command above is summarised by the following table:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! QOS&lt;br /&gt;
! Priority&amp;lt;/br&amp;gt;tier&lt;br /&gt;
! Max&amp;lt;/br&amp;gt;Submit&lt;br /&gt;
! MaxWall&amp;lt;/br&amp;gt;[h]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;CPUs&lt;br /&gt;
! max RAM&amp;lt;/br&amp;gt;[GB]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;3g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;4g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;40GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | build&lt;br /&gt;
| 32&lt;br /&gt;
| 1&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 16&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpulight&lt;br /&gt;
| 8&lt;br /&gt;
| 1&lt;br /&gt;
| 12&lt;br /&gt;
| 2&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | nogpu&lt;br /&gt;
| 4&lt;br /&gt;
| 1&lt;br /&gt;
| 72&lt;br /&gt;
| 16&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpu&lt;br /&gt;
| 2&lt;br /&gt;
| 1&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuwide&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-20&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| 2 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-40&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
A key piece of information in the table above is the &amp;#039;&amp;#039;&amp;#039;priority tier&amp;#039;&amp;#039;&amp;#039; associated to each QOS. In Mufasa 2.0, priority tiers are used to encourage users to use the &amp;#039;&amp;#039;&amp;#039;least powerful QOS that is compatible with their needs&amp;#039;&amp;#039;&amp;#039;, where &amp;quot;powerful&amp;quot; means &amp;quot;rich with resources&amp;quot;. Less powerful QOSes increase the priority of the jobs that use them, so these jobs tend to be executed sooner. See [[#Job priority|Job priority]] for details about how priority affects the execution order of jobs in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
== The &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS ==&lt;br /&gt;
&lt;br /&gt;
This QOS is specifically designed to be used by Mufasa users to &amp;#039;&amp;#039;&amp;#039;quickly build [[System#Containers|container images]]&amp;#039;&amp;#039;&amp;#039;. Its associated priority tier is very high: this, combined with the fact that jobs using this QOS require few resources, means that such jobs usually get executed very soon.&lt;br /&gt;
&lt;br /&gt;
On the other hand, the limited resources and the lack of access to the GPUs make the &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS unsuitable for other tasks.&lt;br /&gt;
&lt;br /&gt;
See [[Singularity#Building Singularity images|Building Singularity images]] for directions about building Singularity container images.&lt;br /&gt;
&lt;br /&gt;
== Restricted QOSes ==&lt;br /&gt;
&lt;br /&gt;
In Mufasa, the most powerful QOSes are reserved to researchers (called &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users: these include academic personnel and Ph.D. students). M.Sc. students, called &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users, cannot use them while running jobs.&lt;br /&gt;
&lt;br /&gt;
See [[#research users and students users|below]] to understand how &amp;#039;&amp;#039;user categories&amp;#039;&amp;#039; work in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
= &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users =&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa belong to two &amp;#039;&amp;#039;&amp;#039;user categories&amp;#039;&amp;#039;&amp;#039;. Categories provide users with different access to  to Mufasa&amp;#039;s resources: the idea being to provide researchers with more access while still letting students benefit from the server&amp;#039;s capabilities.&lt;br /&gt;
&lt;br /&gt;
User categories in Mufasa are the following:&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. academic personnel and Ph.D. students&lt;br /&gt;
::: * can use all QOSes, including the the [[#Restricted QOSes|restricted ones]]&lt;br /&gt;
::: * their jobs have a higher &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a higher number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. M.Sc. students&lt;br /&gt;
::: * cannot use the [[#Restricted QOSes|restricted QOSes]]&lt;br /&gt;
::: * their jobs have a lower &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a lower number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
You can inspect the differences in priority and running jobs between &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association format=&amp;quot;account,priority,maxjobs&amp;quot; | grep -E &amp;#039;Account|research|students&amp;#039;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
   Account   Priority MaxJobs &lt;br /&gt;
  research          4       2 &lt;br /&gt;
  students          1       1&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
To know what limits apply to your own user, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association where user=$USER format=&amp;quot;user,priority,maxjobs,qos%-60&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
      User   Priority MaxJobs QOS                                                          &lt;br /&gt;
---------- ---------- ------- ------------------------------------------------------------ &lt;br /&gt;
    preali          4       2 build,gpu,gpuheavy-20,gpuheavy-40,gpulight,gpuwide,nogpu&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The list under &amp;quot;QOS&amp;quot; shows what QOSes your user is allowed to use when [[User Jobs|running jobs]]. &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users can use all of them, while &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users can only access a subset of them.&lt;br /&gt;
&lt;br /&gt;
= Limits on jobs by the same user =&lt;br /&gt;
&lt;br /&gt;
Mufasa enforces limits on the number of jobs from a single user. Such limits aim at preventing users from &amp;quot;hogging&amp;quot; system resources, and apply to:&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;submitted jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that the user asked SLURM to execute, each of which may currently be either running or queued&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;running jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that are currently in execution&lt;br /&gt;
&lt;br /&gt;
Limits on submitted jobs are set via [[#SLURM Quality of Service (QOS)|QOSes]], while limits on running jobs are set via [[#research users and students users|user category]].&lt;br /&gt;
&lt;br /&gt;
The following table summarises what limits exist:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! &lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;running&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;submitted&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | global limits&amp;lt;br/&amp;gt;(system-wide)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to limits set by individual QOSes (below)&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | limits for&amp;lt;br/&amp;gt;individual&amp;lt;/br&amp;gt;QOSes&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to global limits (above)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for the&amp;#039;&amp;#039; &amp;lt;code&amp;gt;gpuwide&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;QOS&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for each of the other QOSes&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
= Job priority =&lt;br /&gt;
&lt;br /&gt;
Once the execution of a job has been requested, the job is not run immediately: it is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039; by SLURM, together with all the other jobs awaiting execution. The job on top of the queue at any time is the first to be put into execution as soon as the resources it requires are available. &lt;br /&gt;
&lt;br /&gt;
The order of the items in the job queue depends on their &amp;#039;&amp;#039;&amp;#039;priority&amp;#039;&amp;#039;&amp;#039;. The job with the highest priority is put by SLURM on top of the queue, while all other jobs are put in the queue in order of descending priority.&lt;br /&gt;
&lt;br /&gt;
The goal of SLURM is to maximise resource availability: i.e., to ensure the shorter possible wait time before job execution. To achieve this goal, in Mufasa SLURM is configured to &amp;#039;&amp;#039;&amp;#039;encourage users not to ask for resources or execution time that their job doesn&amp;#039;t need&amp;#039;&amp;#039;&amp;#039;. This is done via the priority mechanism: the more resources and/or time a job requests, the lower its priority will be; and the later it will be executed.&lt;br /&gt;
&lt;br /&gt;
Priority management in Mufasa is designed to set up a &amp;#039;&amp;#039;&amp;#039;virtuous cycle&amp;#039;&amp;#039;&amp;#039; where users, by carefully choosing what to ask for, obtain two results:&lt;br /&gt;
* they ensure that their job is executed as soon as possible;&lt;br /&gt;
* they leave as much as possible of Mufasa&amp;#039;s resources free for other users&amp;#039;s jobs.&lt;br /&gt;
&lt;br /&gt;
== Elements determining job priority ==&lt;br /&gt;
In Mufasa, the priority of a job is computed by SLURM according to the following elements:&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#research users and students users|User category]]&amp;#039;&amp;#039;&amp;#039; (i.e., &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;)&lt;br /&gt;
::: Used to provide higher priority to jobs run by &amp;#039;&amp;#039;&amp;#039;research personnel&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#SLURM Quality of Service (QOS)|QOS]]&amp;#039;&amp;#039;&amp;#039; used by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;less resources&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Number of CPUs&amp;#039;&amp;#039;&amp;#039; requested by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;a lower number of CPUs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job Age&amp;#039;&amp;#039;&amp;#039;, i.e. the time that the job has been waiting in the queue&lt;br /&gt;
::: Used to provide higher priority to jobs which have been &amp;#039;&amp;#039;&amp;#039;queued for a longer time&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;FairShare&amp;#039;&amp;#039;&amp;#039;, i.e. a factor computed by SLURM to balance use of the system by different users&lt;br /&gt;
::: Used to provide higher priority to jobs by users who &amp;#039;&amp;#039;&amp;#039;used Mufasa&amp;#039;s resources less than others&amp;#039;&amp;#039;&amp;#039; (see [[#How FairShare works|below]])&lt;br /&gt;
&lt;br /&gt;
You can understand how each of these elements contributes to the priority of the jobs waiting for execution (both yours and of other users) with command &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;. A tailored version of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; that only displays the parameters listed above is the following:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sprio -o &amp;quot;%6i %9u %.5Y %.10B %.5Q %.9T %.5A %.5F&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Important!&lt;br /&gt;
:: The output of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; is only relevant for jobs &amp;#039;&amp;#039;&amp;#039;not yet in execution&amp;#039;&amp;#039;&amp;#039;. For an already running job, the values shown by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; correspond to the priority that the job would get if computed now, which generally differs from the actual priority.&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Here is an example of the output of the command above:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
JOBID  USER      PRIOR      ASSOC   QOS      TRES   AGE FAIRS&lt;br /&gt;
5774   itombini   3492       2500   750  cpu=-800   906   136&lt;br /&gt;
5780   fbottazzi  2266        625   750  cpu=-600   420  1071&lt;br /&gt;
5783   mzoccola   1766        625   750  cpu=-800   217   974&lt;br /&gt;
5790   amoglia    2717       2500   375  cpu=-800   156   487&lt;br /&gt;
5791   qliu       2232       2500   375  cpu=-800   138    19&lt;br /&gt;
5792   fpastori   2197       2500   375  cpu=-800    84    39&lt;br /&gt;
5793   bagostini  2034        625  1500 cpu=-1200    18  1091&lt;br /&gt;
5794   lpassoni   2517       2500   750  cpu=-800     9    58&lt;br /&gt;
5795   lpassoni   2517       2500   750  cpu=-800     9    58&lt;br /&gt;
5796   adurat     3101       2500   750  cpu=-800     8   643&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
In it, column ASSOC (i.e., association) corresponds to user category, while column TRES corresponds to the number of CPUs. Note that the contribution of CPUs to priority is &amp;#039;&amp;#039;negative&amp;#039;&amp;#039;, i.e. it is a penalty. Column PRIOR provides the overall priority of each job.&lt;br /&gt;
&lt;br /&gt;
=== How FairShare works ===&lt;br /&gt;
In Mufasa 2.0, the FairShare of each user is computed according to these rules:&lt;br /&gt;
# It considers the quantity used by (current and past) user jobs of:&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;CPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of CPUs used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;RAM&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the amount of RAM used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;GPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of GPUs used&lt;br /&gt;
#** note: 40 GB GPUs have double the impact on FairShare than 20 GB GPUs&lt;br /&gt;
# The impact on FairShare of any use of resources is &amp;#039;&amp;#039;&amp;#039;proportional to its duration&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
#* example: using 32 GB of RAM for 48h has the same impact of using 64 GB of RAM for 24h&lt;br /&gt;
# FairShare has a &amp;quot;fading memory&amp;quot;: i.e., resource use has less and less impact on FairShare the farther in the past it is&lt;br /&gt;
#* consequence: over time, the &amp;quot;history&amp;quot; of a user gets forgotten by FairShare&lt;br /&gt;
&lt;br /&gt;
== How to maximise the priority of your jobs ==&lt;br /&gt;
&lt;br /&gt;
Every time you run a SLURM job, follow these guidelines:&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Choose the less powerful QOS compatible with the needs of your job&lt;br /&gt;
:: QOSes with access to less resources have higher base priority&lt;br /&gt;
&lt;br /&gt;
; Only request CPUs and RAM that your job really needs&lt;br /&gt;
:: Usually, code exploits multiple CPUs only if specifically designed to do so: if unsure, only ask for 1 CPU&lt;br /&gt;
:: The fewer resources your job asks for, the less it will wait before they become available&lt;br /&gt;
:: The more CPUs your job asks for, the stronger the penalty to its priority&lt;br /&gt;
:: Asking for fewer CPUs and/or less RAM improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Do not request more time than your jobs needs to complete&lt;br /&gt;
:: Make a worst-case estimate and only ask for that duration&lt;br /&gt;
:: Asking for less of Mufasa&amp;#039;s time improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Debug and test code using less powerful QOSes before running it with more powerful QOSes&lt;br /&gt;
:: Your test jobs will get a higher priority and your FairShare will improve&lt;br /&gt;
&lt;br /&gt;
; Cancel jobs that you don&amp;#039;t need them anymore&lt;br /&gt;
:: Always use [[User_Jobs#Cancelling_a_job_with_scancel|scancel]] to delete completed (or crashed) jobs: your Fairshare will improve&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Protip: it&amp;#039;s a good idea to [[User Jobs#Looking for unused GPUs|check for unused GPUs]] before choosing what to request. Requesting a GPU that is currently idle will help your job get executed sooner.&lt;br /&gt;
&lt;br /&gt;
= SLURM partitions =&lt;br /&gt;
&lt;br /&gt;
Partitions are another mechanism provided by SLURM to create different levels of access to system resources. Since in Mufasa 2.0 access to resources is controlled via [[#SLURM Quality of Service (QOS)|QOSes]], partitions are not very relevant. (This is one of the main differences between Mufasa 1.0 and Mufasa 2.0.)&lt;br /&gt;
&lt;br /&gt;
Note, however, that the default values for some features of SLURM jobs (e.g., duration) are [[#Default values|set by the partition]].&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, there is a single SLURM partition, called &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt;, and all jobs run on it. The state of &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt; can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sinfo -o &amp;quot;%10P %5a %9T %11L %10l&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PARTITION  AVAIL STATE     DEFAULTTIME TIMELIMIT &lt;br /&gt;
jobs*      up    idle      1:00:00     3-00:00:00&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where columns correspond to the following information:&lt;br /&gt;
&lt;br /&gt;
:; PARTITION&lt;br /&gt;
:: name of the partition; the asterisks indicates that it&amp;#039;s the default one&lt;br /&gt;
&lt;br /&gt;
:; AVAIL&lt;br /&gt;
:: state/availability of the partition: see [[#Partition availability|below]]&lt;br /&gt;
&lt;br /&gt;
:; STATE&lt;br /&gt;
:: state (using [https://slurm.schedmd.com/sinfo.html#SECTION_NODE-STATE-CODES these codes])&lt;br /&gt;
:: typical values are &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;mixed&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that some of the resources are busy executing jobs while other are idle, and &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;allocated&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that all of the resources are in use&lt;br /&gt;
&lt;br /&gt;
:; DEFAULTTIME&lt;br /&gt;
:: default runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:; TIMELIMIT&lt;br /&gt;
:: maximum runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
The asterisk at the end of the partition name indicates the default partition, i.e. the one on which jobs which do not ask for a specific partition are run.&lt;br /&gt;
&lt;br /&gt;
Command &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; does not tell you about the &amp;#039;&amp;#039;jobs&amp;#039;&amp;#039; submitted to a partition. This information is obtained, instead, with [[User Jobs#Inspecting jobs with squeue|command &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;]].&lt;br /&gt;
&lt;br /&gt;
== Partition availability ==&lt;br /&gt;
&lt;br /&gt;
The most important information that &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; provides is the &amp;#039;&amp;#039;&amp;#039;availability&amp;#039;&amp;#039;&amp;#039; (also called &amp;#039;&amp;#039;&amp;#039;state&amp;#039;&amp;#039;&amp;#039;) of partitions. This is shown in column &amp;quot;AVAIL&amp;quot;. Possible partition states are:&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is available&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed as soon as resources allow&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is in the process of becoming unavailable (i.e., of entering the &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; state: see below)&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is unavailable&lt;br /&gt;
:: There are no running jobs&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
When a partition goes from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; no harm is done to running jobs. In a normally functioning SLURM system, the passage from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; happens only when no jobs are running on the partition. If (e.g., due to a malfunction) the passage happens with jobs still running, they get killed.&lt;br /&gt;
&lt;br /&gt;
A partition in state &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; requires intervention by a [[Roles|Job Administrator]] to be restored to &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
== Default values ==&lt;br /&gt;
&lt;br /&gt;
The features of SLURM partitions, including the &amp;#039;&amp;#039;&amp;#039;default values&amp;#039;&amp;#039;&amp;#039; which are applied to jobs that do not make explicit requests, can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scontrol show partition&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PartitionName=jobs&lt;br /&gt;
   AllowGroups=ALL AllowAccounts=ALL AllowQos=nogpu,gpulight,gpu,gpuwide,gpuheavy-20,gpuheavy-40&lt;br /&gt;
   AllocNodes=ALL Default=YES QoS=N/A&lt;br /&gt;
   DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO&lt;br /&gt;
   MaxNodes=UNLIMITED MaxTime=3-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED&lt;br /&gt;
   Nodes=gn01&lt;br /&gt;
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO&lt;br /&gt;
   OverTimeLimit=NONE PreemptMode=OFF&lt;br /&gt;
   State=UP TotalCPUs=48 TotalNodes=1 SelectTypeParameters=NONE&lt;br /&gt;
   JobDefaults=(null)&lt;br /&gt;
   DefMemPerNode=4096 MaxMemPerNode=UNLIMITED&lt;br /&gt;
   TRES=cpu=48,mem=1011435M,node=1,billing=49,gres/gpu=13,gres/gpu:3g.20gb=5,gres/gpu:40gb=3,gres/gpu:4g.20gb=5&lt;br /&gt;
   TRESBillingWeights=cpu=1.0,gres/gpu:3g.20gb=6.0,gres/gpu:4g.20gb=6.0,gres/gpu:40gb=6.0,mem=0.05g&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In the output of &amp;lt;code&amp;gt;scontrol show partition&amp;lt;/code&amp;gt; the most relevant default values for Mufasa users are:&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefaultTime&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default execution time assigned to a job run on the partition (e.g., 1 hour)&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefMemPerNode&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default amount of RAM assigned to a job run on the partition (e.g., 4GB)&lt;br /&gt;
&lt;br /&gt;
= System resources subjected to limitations =&lt;br /&gt;
&lt;br /&gt;
In systems based on SLURM like Mufasa, &amp;#039;&amp;#039;&amp;#039;TRES (Trackable RESources)&amp;#039;&amp;#039;&amp;#039; are (from [https://slurm.schedmd.com/tres.html SLURM&amp;#039;s documentation] &amp;quot;&amp;#039;&amp;#039;resources that can be tracked for usage or used to enforce limits against.&amp;#039;&amp;#039;&amp;quot;&lt;br /&gt;
&lt;br /&gt;
TRES include CPUs, RAM and &amp;#039;&amp;#039;&amp;#039;GRES&amp;#039;&amp;#039;&amp;#039;. The last term stands for &amp;#039;&amp;#039;Generic RESources&amp;#039;&amp;#039; that a job may need for its execution. In Mufasa, the only &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; resources are the GPUs.&lt;br /&gt;
&lt;br /&gt;
== &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax ==&lt;br /&gt;
&lt;br /&gt;
To ask SLURM to assign GRES resources (i.e., GPUs) to a job, a special syntax must be used. Precisely, the name of each GPU resource takes the form&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:Name:Type&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
Considering the [[System#CPUs and GPUs|GPU complement of Mufasa]], &amp;lt;code&amp;gt;Type&amp;lt;/code&amp;gt; takes the following values:&lt;br /&gt;
&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:40gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 40 Gbytes of RAM&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:4g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 4 compute units&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 3 compute units&lt;br /&gt;
&lt;br /&gt;
So, for instance,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
identifies a resource corresponding to a GPU with 20 GB of RAM and 3 compute units.&lt;br /&gt;
&lt;br /&gt;
When asking for a GRES resource (e.g., in an &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command or an &amp;lt;code&amp;gt;SBATCH&amp;lt;/code&amp;gt; directive of an [[User Jobs#Using execution scripts to run jobs|execution script]]), the syntax required by SLURM is&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:&amp;lt;Type&amp;gt;:&amp;lt;Quantity&amp;gt;&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
where &amp;lt;code&amp;gt;Quantity&amp;lt;/code&amp;gt; is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of type &amp;lt;code&amp;gt;4g.20gb&amp;lt;/code&amp;gt; the syntax is&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:4g.20gb:2&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SLURM&amp;#039;s &amp;#039;&amp;#039;generic resources&amp;#039;&amp;#039; are defined in &amp;lt;code&amp;gt;/etc/slurm/gres.conf&amp;lt;/code&amp;gt;. In order to make GPUs available to SLURM&amp;#039;s &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; management, Mufasa makes use of Nvidia&amp;#039;s [https://developer.nvidia.com/nvidia-management-library-nvml NVML library]. For additional information see [https://slurm.schedmd.com/gres.html SLURM&amp;#039;s documentation].&lt;/div&gt;</summary>
		<author><name>GiulioFontana</name></author>
	</entry>
	<entry>
		<id>https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2583</id>
		<title>SLURM</title>
		<link rel="alternate" type="text/html" href="https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2583"/>
		<updated>2026-11-03T14:52:02Z</updated>

		<summary type="html">&lt;p&gt;GiulioFontana: /* Elements determining job priority */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page presents the features of SLURM that are most relevant to Mufasa&amp;#039;s [[Roles|Job Users]]. Job Users can submit jobs for execution, cancel their own jobs, and see other users&amp;#039; jobs (but not intervene on them).&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa &amp;#039;&amp;#039;&amp;#039;must use SLURM&amp;#039;&amp;#039;&amp;#039; to run resource-heavy processes, i.e. computing jobs that require one or more of the following:&lt;br /&gt;
* GPUs&lt;br /&gt;
* multiple CPUs&lt;br /&gt;
* powerful CPUs&lt;br /&gt;
* a significant amount of RAM&lt;br /&gt;
&lt;br /&gt;
In fact, only processes run via SLURM have access to all the resources of Mufasa. Processes run outside SLURM are executed by the [[System#Login server|login server]] virtual machine, which has minimal resources and no GPUs. Using SLURM is therefore the only way to execute resource-heavy jobs on Mufasa (this is a key difference between Mufasa 1.0 and Mufasa 2.0).&lt;br /&gt;
&lt;br /&gt;
= SLURM in a nutshell =&lt;br /&gt;
&lt;br /&gt;
Computation jobs on Mufasa needs to be launched via [[System#The SLURM job scheduling system|SLURM]]. SLURM provides jobs with access to the [[#System resources subjected to limitations|physical resources]] of Mufasa, such as CPUs, GPUs and RAM. Thanks to SLURM, processing jobs share system resources, optimising their occupation and availability. &lt;br /&gt;
&lt;br /&gt;
When a user runs a job, the job does not get executed immediately and is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039;. SLURM executes jobs according to their order in the queue: the top job in the queue gets executed as soon as the necessary resources are available, while jobs lower in the queue wait longer. The position of a job in the queue is due to the &amp;#039;&amp;#039;&amp;#039;[[#Job priority|priority]]&amp;#039;&amp;#039;&amp;#039; assigned to it by SLURM, with higher-priority jobs closer to the top. As a general rule:&lt;br /&gt;
&lt;br /&gt;
;: &amp;#039;&amp;#039;&amp;#039;the greater the fraction of Mufasa&amp;#039;s overall resources that a job asks for, the lower the job&amp;#039;s priority will be&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
The priority mechanism is used to encourage users to use Mufasa&amp;#039;s resources (i.e.: GPUs, CPUs, RAM, execution time) in an effective and equitable manner. This page includes a [[#How_to_maximise_the_priority_of_your_jobs|chart explaining how to maximise the priority of your jobs]].&lt;br /&gt;
&lt;br /&gt;
The &amp;#039;&amp;#039;&amp;#039;time&amp;#039;&amp;#039;&amp;#039; available to a job for its execution is controlled by SLURM. When a user requests execution of a job, they must specify the duration of the time slot that the job needs. The job must complete its execution before the end of the requested time slot, because at the end of such slot it gets killed by SLURM.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0 access to system resources is managed via SLURM&amp;#039;s [[#SLURM Quality of Service (QOS)|Quality of Service (QOS)]] mechanism (Mufasa 1.0 used [[#SLURM_partitions|partitions]] instead). While [[User Jobs#Running jobs with SLURM|launching a processing job via SLURM]], the user must always specify the chosen QOS. QOSes differ in the set of resources that they provide access to because each of them is designed to fit a given type of job.&lt;br /&gt;
&lt;br /&gt;
Mufasa sets limits to the number of jobs by the same user. This page includes a [[#Limits on jobs by the same user|table summarising such limits]].&lt;br /&gt;
&lt;br /&gt;
= SLURM Quality of Service (QOS) =&lt;br /&gt;
&lt;br /&gt;
Through &amp;#039;&amp;#039;&amp;#039;Quality of Services&amp;#039;&amp;#039;&amp;#039; (&amp;#039;&amp;#039;&amp;#039;QOSes&amp;#039;&amp;#039;&amp;#039;), SLURM lets system configurators assign a name to a set of related constraints.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, QOSes are used to define different levels of access to the server&amp;#039;s resources. When [[User Jobs|executing a job with SLURM]], a user must always &amp;#039;&amp;#039;&amp;#039;specify the QOS&amp;#039;&amp;#039;&amp;#039; that their job will use: this choice, in turn, determines what resources the job is able to access and influences the [[#Job priority|priority]] of the job.&lt;br /&gt;
&lt;br /&gt;
Mufasa&amp;#039;s QOSes and their features can be inspected with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list qos format=name%-11,priority,MaxSubmit,maxwall,maxtres%-80,mintres%-18&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following (see below for the same data in table form):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
Name          Priority MaxSubmit     MaxWall MaxTRES                                                                          MinTRES            &lt;br /&gt;
----------- ---------- --------- ----------- -------------------------------------------------------------------------------- ------------------ &lt;br /&gt;
normal               0                                                                                                                           &lt;br /&gt;
nogpu                4         1  3-00:00:00 cpu=16,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=128G                    &lt;br /&gt;
gpuheavy-20          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=2,mem=128G             gres/gpu:4g.20gb=1 &lt;br /&gt;
gpuheavy-40          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=1,gres/gpu:4g.20gb=0,mem=128G             gres/gpu:40gb=1    &lt;br /&gt;
gpulight             8         1    12:00:00 cpu=2,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpu                  2         1  1-00:00:00 cpu=8,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpuwide              2         2  1-00:00:00 cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=1,mem=64G              gres/gpu:4g.20gb=1 &lt;br /&gt;
build               32         1    02:00:00 cpu=2,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=16G&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The columns of this output are the following:&lt;br /&gt;
&lt;br /&gt;
:; Name&lt;br /&gt;
:: name of the QOS&lt;br /&gt;
&lt;br /&gt;
:; Priority&lt;br /&gt;
:: priority tier associated to the QOS (higher value = higher priority): see [[#Job priority|Job priority]] for details&lt;br /&gt;
&lt;br /&gt;
:; MaxSubmit&lt;br /&gt;
:: maximum number of jobs from a single user that can be submitted to SLURM with this QOS; submitted jobs include both running and queued jobs&lt;br /&gt;
:: See [[#Limits on jobs by the same user|Limits on jobs by the same user]] for an overview of the limits on jobs set by Mufasa.&lt;br /&gt;
&lt;br /&gt;
:; MaxWall&lt;br /&gt;
:: maximum wall clock duration of the jobs using the QOS (after which they are killed by SLURM), in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
:: For some QOSes these are not set: it means that they are determined by the [[#SLURM partitions|partition]]. Partitions also define the [[#Default values|default duration]] of jobs.&lt;br /&gt;
&lt;br /&gt;
:; MaxTRES&lt;br /&gt;
:: amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) available to a job using the QOS, where&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;cpu=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of CPUs (i.e., processor cores) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of CPUs specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;gres/&amp;#039;&amp;#039;gpu:Type&amp;#039;&amp;#039;=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of GPUs of class &amp;lt;code&amp;gt;&amp;#039;&amp;#039;Type&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; (see [[User Jobs#gres syntax|&amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax]]) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; (for QOSes that allow access to GPUs) if not specified, the job cannot be launched&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;mem=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;G&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum amount of system RAM is &amp;#039;&amp;#039;K&amp;#039;&amp;#039; GBytes&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of RAM specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
&lt;br /&gt;
:; MinTRES&lt;br /&gt;
:: minimum amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) that a job using the QOS must request in order to actually get executed by SLURM.&lt;br /&gt;
:: (If your job does not actually need these resources, you&amp;#039;ve chosen the wrong QOS: you can use one with a higher priority.)&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; QOS is the default one, and exists only to ensure that users always specify a QOS when running a job. Since &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; has zero priority and no resources, a job run using this QOS would never be run.&lt;br /&gt;
&lt;br /&gt;
The information provided by the &amp;lt;code&amp;gt;sacctmgr list qos&amp;lt;/code&amp;gt; command above is summarised by the following table:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! QOS&lt;br /&gt;
! Priority&amp;lt;/br&amp;gt;tier&lt;br /&gt;
! Max&amp;lt;/br&amp;gt;Submit&lt;br /&gt;
! MaxWall&amp;lt;/br&amp;gt;[h]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;CPUs&lt;br /&gt;
! max RAM&amp;lt;/br&amp;gt;[GB]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;3g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;4g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;40GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | build&lt;br /&gt;
| 32&lt;br /&gt;
| 1&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 16&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpulight&lt;br /&gt;
| 8&lt;br /&gt;
| 1&lt;br /&gt;
| 12&lt;br /&gt;
| 2&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | nogpu&lt;br /&gt;
| 4&lt;br /&gt;
| 1&lt;br /&gt;
| 72&lt;br /&gt;
| 16&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpu&lt;br /&gt;
| 2&lt;br /&gt;
| 1&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuwide&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-20&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| 2 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-40&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
A key piece of information in the table above is the &amp;#039;&amp;#039;&amp;#039;priority tier&amp;#039;&amp;#039;&amp;#039; associated to each QOS. In Mufasa 2.0, priority tiers are used to encourage users to use the &amp;#039;&amp;#039;&amp;#039;least powerful QOS that is compatible with their needs&amp;#039;&amp;#039;&amp;#039;, where &amp;quot;powerful&amp;quot; means &amp;quot;rich with resources&amp;quot;. Less powerful QOSes increase the priority of the jobs that use them, so these jobs tend to be executed sooner. See [[#Job priority|Job priority]] for details about how priority affects the execution order of jobs in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
== The &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS ==&lt;br /&gt;
&lt;br /&gt;
This QOS is specifically designed to be used by Mufasa users to &amp;#039;&amp;#039;&amp;#039;quickly build [[System#Containers|container images]]&amp;#039;&amp;#039;&amp;#039;. Its associated priority tier is very high: this, combined with the fact that jobs using this QOS require few resources, means that such jobs usually get executed very soon.&lt;br /&gt;
&lt;br /&gt;
On the other hand, the limited resources and the lack of access to the GPUs make the &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS unsuitable for other tasks.&lt;br /&gt;
&lt;br /&gt;
See [[Singularity#Building Singularity images|Building Singularity images]] for directions about building Singularity container images.&lt;br /&gt;
&lt;br /&gt;
== Restricted QOSes ==&lt;br /&gt;
&lt;br /&gt;
In Mufasa, the most powerful QOSes are reserved to researchers (called &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users: these include academic personnel and Ph.D. students). M.Sc. students, called &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users, cannot use them while running jobs.&lt;br /&gt;
&lt;br /&gt;
See [[#research users and students users|below]] to understand how &amp;#039;&amp;#039;user categories&amp;#039;&amp;#039; work in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
= &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users =&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa belong to two &amp;#039;&amp;#039;&amp;#039;user categories&amp;#039;&amp;#039;&amp;#039;. Categories provide users with different access to  to Mufasa&amp;#039;s resources: the idea being to provide researchers with more access while still letting students benefit from the server&amp;#039;s capabilities.&lt;br /&gt;
&lt;br /&gt;
User categories in Mufasa are the following:&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. academic personnel and Ph.D. students&lt;br /&gt;
::: * can use all QOSes, including the the [[#Restricted QOSes|restricted ones]]&lt;br /&gt;
::: * their jobs have a higher &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a higher number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. M.Sc. students&lt;br /&gt;
::: * cannot use the [[#Restricted QOSes|restricted QOSes]]&lt;br /&gt;
::: * their jobs have a lower &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a lower number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
You can inspect the differences in priority and running jobs between &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association format=&amp;quot;account,priority,maxjobs&amp;quot; | grep -E &amp;#039;Account|research|students&amp;#039;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
   Account   Priority MaxJobs &lt;br /&gt;
  research          4       2 &lt;br /&gt;
  students          1       1&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
To know what limits apply to your own user, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association where user=$USER format=&amp;quot;user,priority,maxjobs,qos%-60&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
      User   Priority MaxJobs QOS                                                          &lt;br /&gt;
---------- ---------- ------- ------------------------------------------------------------ &lt;br /&gt;
    preali          4       2 build,gpu,gpuheavy-20,gpuheavy-40,gpulight,gpuwide,nogpu&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The list under &amp;quot;QOS&amp;quot; shows what QOSes your user is allowed to use when [[User Jobs|running jobs]]. &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users can use all of them, while &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users can only access a subset of them.&lt;br /&gt;
&lt;br /&gt;
= Limits on jobs by the same user =&lt;br /&gt;
&lt;br /&gt;
Mufasa enforces limits on the number of jobs from a single user. Such limits aim at preventing users from &amp;quot;hogging&amp;quot; system resources, and apply to:&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;submitted jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that the user asked SLURM to execute, each of which may currently be either running or queued&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;running jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that are currently in execution&lt;br /&gt;
&lt;br /&gt;
Limits on submitted jobs are set via [[#SLURM Quality of Service (QOS)|QOSes]], while limits on running jobs are set via [[#research users and students users|user category]].&lt;br /&gt;
&lt;br /&gt;
The following table summarises what limits exist:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! &lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;running&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;submitted&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | global limits&amp;lt;br/&amp;gt;(system-wide)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to limits set by individual QOSes (below)&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | limits for&amp;lt;br/&amp;gt;individual&amp;lt;/br&amp;gt;QOSes&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to global limits (above)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for the&amp;#039;&amp;#039; &amp;lt;code&amp;gt;gpuwide&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;QOS&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for each of the other QOSes&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
= Job priority =&lt;br /&gt;
&lt;br /&gt;
Once the execution of a job has been requested, the job is not run immediately: it is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039; by SLURM, together with all the other jobs awaiting execution. The job on top of the queue at any time is the first to be put into execution as soon as the resources it requires are available. &lt;br /&gt;
&lt;br /&gt;
The order of the items in the job queue depends on their &amp;#039;&amp;#039;&amp;#039;priority&amp;#039;&amp;#039;&amp;#039;. The job with the highest priority is put by SLURM on top of the queue, while all other jobs are put in the queue in order of descending priority.&lt;br /&gt;
&lt;br /&gt;
The goal of SLURM is to maximise resource availability: i.e., to ensure the shorter possible wait time before job execution. To achieve this goal, in Mufasa SLURM is configured to &amp;#039;&amp;#039;&amp;#039;encourage users not to ask for resources or execution time that their job doesn&amp;#039;t need&amp;#039;&amp;#039;&amp;#039;. This is done via the priority mechanism: the more resources and/or time a job requests, the lower its priority will be; and the later it will be executed.&lt;br /&gt;
&lt;br /&gt;
Priority management in Mufasa is designed to set up a &amp;#039;&amp;#039;&amp;#039;virtuous cycle&amp;#039;&amp;#039;&amp;#039; where users, by carefully choosing what to ask for, obtain two results:&lt;br /&gt;
* they ensure that their job is executed as soon as possible;&lt;br /&gt;
* they leave as much as possible of Mufasa&amp;#039;s resources free for other users&amp;#039;s jobs.&lt;br /&gt;
&lt;br /&gt;
== Elements determining job priority ==&lt;br /&gt;
In Mufasa, the priority of a job is computed by SLURM according to the following elements:&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#research users and students users|User category]]&amp;#039;&amp;#039;&amp;#039; (i.e., &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;)&lt;br /&gt;
::: Used to provide higher priority to jobs run by &amp;#039;&amp;#039;&amp;#039;research personnel&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#SLURM Quality of Service (QOS)|QOS]]&amp;#039;&amp;#039;&amp;#039; used by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;less resources&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Number of CPUs&amp;#039;&amp;#039;&amp;#039; requested by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;a lower number of CPUs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job Age&amp;#039;&amp;#039;&amp;#039;, i.e. the time that the job has been waiting in the queue&lt;br /&gt;
::: Used to provide higher priority to jobs which have been &amp;#039;&amp;#039;&amp;#039;queued for a longer time&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;FairShare&amp;#039;&amp;#039;&amp;#039;, i.e. a factor computed by SLURM to balance use of the system by different users&lt;br /&gt;
::: Used to provide higher priority to jobs by users who &amp;#039;&amp;#039;&amp;#039;used Mufasa&amp;#039;s resources less than others&amp;#039;&amp;#039;&amp;#039; (see [[#How FairShare works|below]])&lt;br /&gt;
&lt;br /&gt;
You can understand how each of these elements contributes to the priority of the jobs waiting for execution (both yours and of other users) with command &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;. A tailored version of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; that only displays the parameters listed above is the following:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sprio -o &amp;quot;%6i %9u %.5Y %.10B %.5Q %.9T %.5A %.5F&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Important!&lt;br /&gt;
:: The output of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; is only relevant for jobs &amp;#039;&amp;#039;&amp;#039;not yet in execution&amp;#039;&amp;#039;&amp;#039;. For an already running job, the values shown by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; correspond to the priority that the job would get if computed now, which generally differs from the actual priority.&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Here is an example of the output of the command above:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
JOBID  USER      PRIOR      ASSOC   QOS      TRES   AGE FAIRS&lt;br /&gt;
5774   itombini   3492       2500   750  cpu=-800   906   136&lt;br /&gt;
5780   fbottazzi  2266        625   750  cpu=-600   420  1071&lt;br /&gt;
5783   mzoccola   1766        625   750  cpu=-800   217   974&lt;br /&gt;
5790   amoglia    2717       2500   375  cpu=-800   156   487&lt;br /&gt;
5791   qliu       2232       2500   375  cpu=-800   138    19&lt;br /&gt;
5792   fpastori   2197       2500   375  cpu=-800    84    39&lt;br /&gt;
5793   bagostini  2034        625  1500 cpu=-1200    18  1091&lt;br /&gt;
5794   lpassoni   2517       2500   750  cpu=-800     9    58&lt;br /&gt;
5795   lpassoni   2517       2500   750  cpu=-800     9    58&lt;br /&gt;
5796   adurat     3101       2500   750  cpu=-800     8   643&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
In it, column ASSOC (i.e., association) corresponds to user category, while column TRES corresponds to the number of CPUs. Note that the contribution of CPUs to priority is &amp;#039;&amp;#039;negative&amp;#039;&amp;#039;, i.e. it is a penalty. Column PRIOR provides the overall priority of each job.&lt;br /&gt;
&lt;br /&gt;
=== How FairShare works ===&lt;br /&gt;
In Mufasa 2.0, the FairShare of each user is computed according to these rules:&lt;br /&gt;
# It considers the quantity used by (current and past) user jobs of:&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;CPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of CPUs used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;RAM&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the amount of RAM used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;GPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of GPUs used&lt;br /&gt;
#** note: 40 GB GPUs have double the impact on FairShare than 20 GB GPUs&lt;br /&gt;
# The impact on FairShare of any use of resources is &amp;#039;&amp;#039;&amp;#039;proportional to its duration&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
#* example: using 32 GB of RAM for 48h has the same impact of using 64 GB of RAM for 24h&lt;br /&gt;
# FairShare has a &amp;quot;fading memory&amp;quot;: i.e., resource use has less and less impact on FairShare the farther in the past it is&lt;br /&gt;
#* consequence: over time, the &amp;quot;history&amp;quot; of a user gets forgotten by FairShare&lt;br /&gt;
&lt;br /&gt;
== How to maximise the priority of your jobs ==&lt;br /&gt;
&lt;br /&gt;
Every time you run a SLURM job, follow these guidelines:&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Choose the less powerful QOS compatible with the needs of your job&lt;br /&gt;
:: QOSes with access to less resources have higher base priority&lt;br /&gt;
&lt;br /&gt;
; Only request CPUs and RAM that your job really needs&lt;br /&gt;
:: Usually, code exploits multiple CPUs only if specifically designed to do so: if unsure, only ask for 1 CPU&lt;br /&gt;
:: The fewer resources your job asks for, the less it will wait before they become available&lt;br /&gt;
:: The more CPUs your job asks for, the stronger the penalty to its priority&lt;br /&gt;
:: Asking for fewer CPUs and/or less RAM improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Do not request more time than your jobs needs to complete&lt;br /&gt;
:: Make a worst-case estimate and only ask for that duration&lt;br /&gt;
:: Asking for less of Mufasa&amp;#039;s time improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Debug and test code using less powerful QOSes before running it with more powerful QOSes&lt;br /&gt;
:: Your test jobs will get a higher priority and your FairShare will improve&lt;br /&gt;
&lt;br /&gt;
; Cancel jobs that you don&amp;#039;t need them anymore&lt;br /&gt;
:: Always use [[User_Jobs#Cancelling_a_job_with_scancel|scancel]] to delete completed (or crashed) jobs: your Fairshare will improve&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Protip: it&amp;#039;s a good idea to [[User Jobs#Looking for unused GPUs|check for unused GPUs]] before choosing what to request. Requesting a GPU that is currently idle will help your job get executed sooner.&lt;br /&gt;
&lt;br /&gt;
= SLURM partitions =&lt;br /&gt;
&lt;br /&gt;
Partitions are another mechanism provided by SLURM to create different levels of access to system resources. Since in Mufasa 2.0 access to resources is controlled via [[#SLURM Quality of Service (QOS)|QOSes]], partitions are not very relevant. (This is one of the main differences between Mufasa 1.0 and Mufasa 2.0.)&lt;br /&gt;
&lt;br /&gt;
Note, however, that the default values for some features of SLURM jobs (e.g., duration) are [[#Default values|set by the partition]].&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, there is a single SLURM partition, called &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt;, and all jobs run on it. The state of &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt; can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sinfo -o &amp;quot;%10P %5a %9T %11L %10l&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PARTITION  AVAIL STATE     DEFAULTTIME TIMELIMIT &lt;br /&gt;
jobs*      up    idle      1:00:00     3-00:00:00&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where columns correspond to the following information:&lt;br /&gt;
&lt;br /&gt;
:; PARTITION&lt;br /&gt;
:: name of the partition; the asterisks indicates that it&amp;#039;s the default one&lt;br /&gt;
&lt;br /&gt;
:; AVAIL&lt;br /&gt;
:: state/availability of the partition: see [[#Partition availability|below]]&lt;br /&gt;
&lt;br /&gt;
:; STATE&lt;br /&gt;
:: state (using [https://slurm.schedmd.com/sinfo.html#SECTION_NODE-STATE-CODES these codes])&lt;br /&gt;
:: typical values are &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;mixed&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that some of the resources are busy executing jobs while other are idle, and &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;allocated&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that all of the resources are in use&lt;br /&gt;
&lt;br /&gt;
:; DEFAULTTIME&lt;br /&gt;
:: default runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:; TIMELIMIT&lt;br /&gt;
:: maximum runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
The asterisk at the end of the partition name indicates the default partition, i.e. the one on which jobs which do not ask for a specific partition are run.&lt;br /&gt;
&lt;br /&gt;
Command &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; does not tell you about the &amp;#039;&amp;#039;jobs&amp;#039;&amp;#039; submitted to a partition. This information is obtained, instead, with [[User Jobs#Inspecting jobs with squeue|command &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;]].&lt;br /&gt;
&lt;br /&gt;
== Partition availability ==&lt;br /&gt;
&lt;br /&gt;
The most important information that &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; provides is the &amp;#039;&amp;#039;&amp;#039;availability&amp;#039;&amp;#039;&amp;#039; (also called &amp;#039;&amp;#039;&amp;#039;state&amp;#039;&amp;#039;&amp;#039;) of partitions. This is shown in column &amp;quot;AVAIL&amp;quot;. Possible partition states are:&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is available&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed as soon as resources allow&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is in the process of becoming unavailable (i.e., of entering the &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; state: see below)&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is unavailable&lt;br /&gt;
:: There are no running jobs&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
When a partition goes from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; no harm is done to running jobs. In a normally functioning SLURM system, the passage from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; happens only when no jobs are running on the partition. If (e.g., due to a malfunction) the passage happens with jobs still running, they get killed.&lt;br /&gt;
&lt;br /&gt;
A partition in state &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; requires intervention by a [[Roles|Job Administrator]] to be restored to &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
== Default values ==&lt;br /&gt;
&lt;br /&gt;
The features of SLURM partitions, including the &amp;#039;&amp;#039;&amp;#039;default values&amp;#039;&amp;#039;&amp;#039; which are applied to jobs that do not make explicit requests, can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scontrol show partition&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PartitionName=jobs&lt;br /&gt;
   AllowGroups=ALL AllowAccounts=ALL AllowQos=nogpu,gpulight,gpu,gpuwide,gpuheavy-20,gpuheavy-40&lt;br /&gt;
   AllocNodes=ALL Default=YES QoS=N/A&lt;br /&gt;
=&amp;gt; DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO&lt;br /&gt;
   MaxNodes=UNLIMITED MaxTime=3-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED&lt;br /&gt;
   Nodes=gn01&lt;br /&gt;
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO&lt;br /&gt;
   OverTimeLimit=NONE PreemptMode=OFF&lt;br /&gt;
   State=UP TotalCPUs=48 TotalNodes=1 SelectTypeParameters=NONE&lt;br /&gt;
   JobDefaults=(null)&lt;br /&gt;
=&amp;gt; DefMemPerNode=4096 MaxMemPerNode=UNLIMITED&lt;br /&gt;
   TRES=cpu=48,mem=1011435M,node=1,billing=49,gres/gpu=13,gres/gpu:3g.20gb=5,gres/gpu:40gb=3,gres/gpu:4g.20gb=5&lt;br /&gt;
   TRESBillingWeights=cpu=1.0,gres/gpu:3g.20gb=6.0,gres/gpu:4g.20gb=6.0,gres/gpu:40gb=6.0,mem=0.05g&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In the example, we have highlighted with  &amp;#039;&amp;#039;&amp;#039;=&amp;gt;&amp;#039;&amp;#039;&amp;#039;  the most relevant default values for Mufasa users, i.e.:&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefaultTime&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default execution time assigned to a job run on the partition (e.g., 1 hour)&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefMemPerNode&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default amount of RAM assigned to a job run on the partition (e.g., 4GB)&lt;br /&gt;
&lt;br /&gt;
= System resources subjected to limitations =&lt;br /&gt;
&lt;br /&gt;
In systems based on SLURM like Mufasa, &amp;#039;&amp;#039;&amp;#039;TRES (Trackable RESources)&amp;#039;&amp;#039;&amp;#039; are (from [https://slurm.schedmd.com/tres.html SLURM&amp;#039;s documentation] &amp;quot;&amp;#039;&amp;#039;resources that can be tracked for usage or used to enforce limits against.&amp;#039;&amp;#039;&amp;quot;&lt;br /&gt;
&lt;br /&gt;
TRES include CPUs, RAM and &amp;#039;&amp;#039;&amp;#039;GRES&amp;#039;&amp;#039;&amp;#039;. The last term stands for &amp;#039;&amp;#039;Generic RESources&amp;#039;&amp;#039; that a job may need for its execution. In Mufasa, the only &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; resources are the GPUs.&lt;br /&gt;
&lt;br /&gt;
== &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax ==&lt;br /&gt;
&lt;br /&gt;
To ask SLURM to assign GRES resources (i.e., GPUs) to a job, a special syntax must be used. Precisely, the name of each GPU resource takes the form&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:Name:Type&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
Considering the [[System#CPUs and GPUs|GPU complement of Mufasa]], &amp;lt;code&amp;gt;Type&amp;lt;/code&amp;gt; takes the following values:&lt;br /&gt;
&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:40gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 40 Gbytes of RAM&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:4g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 4 compute units&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 3 compute units&lt;br /&gt;
&lt;br /&gt;
So, for instance,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
identifies a resource corresponding to a GPU with 20 GB of RAM and 3 compute units.&lt;br /&gt;
&lt;br /&gt;
When asking for a GRES resource (e.g., in an &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command or an &amp;lt;code&amp;gt;SBATCH&amp;lt;/code&amp;gt; directive of an [[User Jobs#Using execution scripts to run jobs|execution script]]), the syntax required by SLURM is&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:&amp;lt;Type&amp;gt;:&amp;lt;Quantity&amp;gt;&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
where &amp;lt;code&amp;gt;Quantity&amp;lt;/code&amp;gt; is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of type &amp;lt;code&amp;gt;4g.20gb&amp;lt;/code&amp;gt; the syntax is&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:4g.20gb:2&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SLURM&amp;#039;s &amp;#039;&amp;#039;generic resources&amp;#039;&amp;#039; are defined in &amp;lt;code&amp;gt;/etc/slurm/gres.conf&amp;lt;/code&amp;gt;. In order to make GPUs available to SLURM&amp;#039;s &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; management, Mufasa makes use of Nvidia&amp;#039;s [https://developer.nvidia.com/nvidia-management-library-nvml NVML library]. For additional information see [https://slurm.schedmd.com/gres.html SLURM&amp;#039;s documentation].&lt;/div&gt;</summary>
		<author><name>GiulioFontana</name></author>
	</entry>
	<entry>
		<id>https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2582</id>
		<title>SLURM</title>
		<link rel="alternate" type="text/html" href="https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2582"/>
		<updated>2026-11-03T14:48:26Z</updated>

		<summary type="html">&lt;p&gt;GiulioFontana: /* How to maximise the priority of your jobs */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page presents the features of SLURM that are most relevant to Mufasa&amp;#039;s [[Roles|Job Users]]. Job Users can submit jobs for execution, cancel their own jobs, and see other users&amp;#039; jobs (but not intervene on them).&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa &amp;#039;&amp;#039;&amp;#039;must use SLURM&amp;#039;&amp;#039;&amp;#039; to run resource-heavy processes, i.e. computing jobs that require one or more of the following:&lt;br /&gt;
* GPUs&lt;br /&gt;
* multiple CPUs&lt;br /&gt;
* powerful CPUs&lt;br /&gt;
* a significant amount of RAM&lt;br /&gt;
&lt;br /&gt;
In fact, only processes run via SLURM have access to all the resources of Mufasa. Processes run outside SLURM are executed by the [[System#Login server|login server]] virtual machine, which has minimal resources and no GPUs. Using SLURM is therefore the only way to execute resource-heavy jobs on Mufasa (this is a key difference between Mufasa 1.0 and Mufasa 2.0).&lt;br /&gt;
&lt;br /&gt;
= SLURM in a nutshell =&lt;br /&gt;
&lt;br /&gt;
Computation jobs on Mufasa needs to be launched via [[System#The SLURM job scheduling system|SLURM]]. SLURM provides jobs with access to the [[#System resources subjected to limitations|physical resources]] of Mufasa, such as CPUs, GPUs and RAM. Thanks to SLURM, processing jobs share system resources, optimising their occupation and availability. &lt;br /&gt;
&lt;br /&gt;
When a user runs a job, the job does not get executed immediately and is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039;. SLURM executes jobs according to their order in the queue: the top job in the queue gets executed as soon as the necessary resources are available, while jobs lower in the queue wait longer. The position of a job in the queue is due to the &amp;#039;&amp;#039;&amp;#039;[[#Job priority|priority]]&amp;#039;&amp;#039;&amp;#039; assigned to it by SLURM, with higher-priority jobs closer to the top. As a general rule:&lt;br /&gt;
&lt;br /&gt;
;: &amp;#039;&amp;#039;&amp;#039;the greater the fraction of Mufasa&amp;#039;s overall resources that a job asks for, the lower the job&amp;#039;s priority will be&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
The priority mechanism is used to encourage users to use Mufasa&amp;#039;s resources (i.e.: GPUs, CPUs, RAM, execution time) in an effective and equitable manner. This page includes a [[#How_to_maximise_the_priority_of_your_jobs|chart explaining how to maximise the priority of your jobs]].&lt;br /&gt;
&lt;br /&gt;
The &amp;#039;&amp;#039;&amp;#039;time&amp;#039;&amp;#039;&amp;#039; available to a job for its execution is controlled by SLURM. When a user requests execution of a job, they must specify the duration of the time slot that the job needs. The job must complete its execution before the end of the requested time slot, because at the end of such slot it gets killed by SLURM.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0 access to system resources is managed via SLURM&amp;#039;s [[#SLURM Quality of Service (QOS)|Quality of Service (QOS)]] mechanism (Mufasa 1.0 used [[#SLURM_partitions|partitions]] instead). While [[User Jobs#Running jobs with SLURM|launching a processing job via SLURM]], the user must always specify the chosen QOS. QOSes differ in the set of resources that they provide access to because each of them is designed to fit a given type of job.&lt;br /&gt;
&lt;br /&gt;
Mufasa sets limits to the number of jobs by the same user. This page includes a [[#Limits on jobs by the same user|table summarising such limits]].&lt;br /&gt;
&lt;br /&gt;
= SLURM Quality of Service (QOS) =&lt;br /&gt;
&lt;br /&gt;
Through &amp;#039;&amp;#039;&amp;#039;Quality of Services&amp;#039;&amp;#039;&amp;#039; (&amp;#039;&amp;#039;&amp;#039;QOSes&amp;#039;&amp;#039;&amp;#039;), SLURM lets system configurators assign a name to a set of related constraints.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, QOSes are used to define different levels of access to the server&amp;#039;s resources. When [[User Jobs|executing a job with SLURM]], a user must always &amp;#039;&amp;#039;&amp;#039;specify the QOS&amp;#039;&amp;#039;&amp;#039; that their job will use: this choice, in turn, determines what resources the job is able to access and influences the [[#Job priority|priority]] of the job.&lt;br /&gt;
&lt;br /&gt;
Mufasa&amp;#039;s QOSes and their features can be inspected with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list qos format=name%-11,priority,MaxSubmit,maxwall,maxtres%-80,mintres%-18&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following (see below for the same data in table form):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
Name          Priority MaxSubmit     MaxWall MaxTRES                                                                          MinTRES            &lt;br /&gt;
----------- ---------- --------- ----------- -------------------------------------------------------------------------------- ------------------ &lt;br /&gt;
normal               0                                                                                                                           &lt;br /&gt;
nogpu                4         1  3-00:00:00 cpu=16,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=128G                    &lt;br /&gt;
gpuheavy-20          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=2,mem=128G             gres/gpu:4g.20gb=1 &lt;br /&gt;
gpuheavy-40          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=1,gres/gpu:4g.20gb=0,mem=128G             gres/gpu:40gb=1    &lt;br /&gt;
gpulight             8         1    12:00:00 cpu=2,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpu                  2         1  1-00:00:00 cpu=8,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpuwide              2         2  1-00:00:00 cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=1,mem=64G              gres/gpu:4g.20gb=1 &lt;br /&gt;
build               32         1    02:00:00 cpu=2,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=16G&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The columns of this output are the following:&lt;br /&gt;
&lt;br /&gt;
:; Name&lt;br /&gt;
:: name of the QOS&lt;br /&gt;
&lt;br /&gt;
:; Priority&lt;br /&gt;
:: priority tier associated to the QOS (higher value = higher priority): see [[#Job priority|Job priority]] for details&lt;br /&gt;
&lt;br /&gt;
:; MaxSubmit&lt;br /&gt;
:: maximum number of jobs from a single user that can be submitted to SLURM with this QOS; submitted jobs include both running and queued jobs&lt;br /&gt;
:: See [[#Limits on jobs by the same user|Limits on jobs by the same user]] for an overview of the limits on jobs set by Mufasa.&lt;br /&gt;
&lt;br /&gt;
:; MaxWall&lt;br /&gt;
:: maximum wall clock duration of the jobs using the QOS (after which they are killed by SLURM), in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
:: For some QOSes these are not set: it means that they are determined by the [[#SLURM partitions|partition]]. Partitions also define the [[#Default values|default duration]] of jobs.&lt;br /&gt;
&lt;br /&gt;
:; MaxTRES&lt;br /&gt;
:: amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) available to a job using the QOS, where&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;cpu=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of CPUs (i.e., processor cores) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of CPUs specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;gres/&amp;#039;&amp;#039;gpu:Type&amp;#039;&amp;#039;=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of GPUs of class &amp;lt;code&amp;gt;&amp;#039;&amp;#039;Type&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; (see [[User Jobs#gres syntax|&amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax]]) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; (for QOSes that allow access to GPUs) if not specified, the job cannot be launched&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;mem=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;G&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum amount of system RAM is &amp;#039;&amp;#039;K&amp;#039;&amp;#039; GBytes&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of RAM specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
&lt;br /&gt;
:; MinTRES&lt;br /&gt;
:: minimum amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) that a job using the QOS must request in order to actually get executed by SLURM.&lt;br /&gt;
:: (If your job does not actually need these resources, you&amp;#039;ve chosen the wrong QOS: you can use one with a higher priority.)&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; QOS is the default one, and exists only to ensure that users always specify a QOS when running a job. Since &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; has zero priority and no resources, a job run using this QOS would never be run.&lt;br /&gt;
&lt;br /&gt;
The information provided by the &amp;lt;code&amp;gt;sacctmgr list qos&amp;lt;/code&amp;gt; command above is summarised by the following table:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! QOS&lt;br /&gt;
! Priority&amp;lt;/br&amp;gt;tier&lt;br /&gt;
! Max&amp;lt;/br&amp;gt;Submit&lt;br /&gt;
! MaxWall&amp;lt;/br&amp;gt;[h]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;CPUs&lt;br /&gt;
! max RAM&amp;lt;/br&amp;gt;[GB]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;3g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;4g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;40GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | build&lt;br /&gt;
| 32&lt;br /&gt;
| 1&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 16&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpulight&lt;br /&gt;
| 8&lt;br /&gt;
| 1&lt;br /&gt;
| 12&lt;br /&gt;
| 2&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | nogpu&lt;br /&gt;
| 4&lt;br /&gt;
| 1&lt;br /&gt;
| 72&lt;br /&gt;
| 16&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpu&lt;br /&gt;
| 2&lt;br /&gt;
| 1&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuwide&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-20&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| 2 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-40&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
A key piece of information in the table above is the &amp;#039;&amp;#039;&amp;#039;priority tier&amp;#039;&amp;#039;&amp;#039; associated to each QOS. In Mufasa 2.0, priority tiers are used to encourage users to use the &amp;#039;&amp;#039;&amp;#039;least powerful QOS that is compatible with their needs&amp;#039;&amp;#039;&amp;#039;, where &amp;quot;powerful&amp;quot; means &amp;quot;rich with resources&amp;quot;. Less powerful QOSes increase the priority of the jobs that use them, so these jobs tend to be executed sooner. See [[#Job priority|Job priority]] for details about how priority affects the execution order of jobs in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
== The &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS ==&lt;br /&gt;
&lt;br /&gt;
This QOS is specifically designed to be used by Mufasa users to &amp;#039;&amp;#039;&amp;#039;quickly build [[System#Containers|container images]]&amp;#039;&amp;#039;&amp;#039;. Its associated priority tier is very high: this, combined with the fact that jobs using this QOS require few resources, means that such jobs usually get executed very soon.&lt;br /&gt;
&lt;br /&gt;
On the other hand, the limited resources and the lack of access to the GPUs make the &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS unsuitable for other tasks.&lt;br /&gt;
&lt;br /&gt;
See [[Singularity#Building Singularity images|Building Singularity images]] for directions about building Singularity container images.&lt;br /&gt;
&lt;br /&gt;
== Restricted QOSes ==&lt;br /&gt;
&lt;br /&gt;
In Mufasa, the most powerful QOSes are reserved to researchers (called &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users: these include academic personnel and Ph.D. students). M.Sc. students, called &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users, cannot use them while running jobs.&lt;br /&gt;
&lt;br /&gt;
See [[#research users and students users|below]] to understand how &amp;#039;&amp;#039;user categories&amp;#039;&amp;#039; work in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
= &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users =&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa belong to two &amp;#039;&amp;#039;&amp;#039;user categories&amp;#039;&amp;#039;&amp;#039;. Categories provide users with different access to  to Mufasa&amp;#039;s resources: the idea being to provide researchers with more access while still letting students benefit from the server&amp;#039;s capabilities.&lt;br /&gt;
&lt;br /&gt;
User categories in Mufasa are the following:&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. academic personnel and Ph.D. students&lt;br /&gt;
::: * can use all QOSes, including the the [[#Restricted QOSes|restricted ones]]&lt;br /&gt;
::: * their jobs have a higher &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a higher number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. M.Sc. students&lt;br /&gt;
::: * cannot use the [[#Restricted QOSes|restricted QOSes]]&lt;br /&gt;
::: * their jobs have a lower &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a lower number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
You can inspect the differences in priority and running jobs between &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association format=&amp;quot;account,priority,maxjobs&amp;quot; | grep -E &amp;#039;Account|research|students&amp;#039;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
   Account   Priority MaxJobs &lt;br /&gt;
  research          4       2 &lt;br /&gt;
  students          1       1&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
To know what limits apply to your own user, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association where user=$USER format=&amp;quot;user,priority,maxjobs,qos%-60&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
      User   Priority MaxJobs QOS                                                          &lt;br /&gt;
---------- ---------- ------- ------------------------------------------------------------ &lt;br /&gt;
    preali          4       2 build,gpu,gpuheavy-20,gpuheavy-40,gpulight,gpuwide,nogpu&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The list under &amp;quot;QOS&amp;quot; shows what QOSes your user is allowed to use when [[User Jobs|running jobs]]. &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users can use all of them, while &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users can only access a subset of them.&lt;br /&gt;
&lt;br /&gt;
= Limits on jobs by the same user =&lt;br /&gt;
&lt;br /&gt;
Mufasa enforces limits on the number of jobs from a single user. Such limits aim at preventing users from &amp;quot;hogging&amp;quot; system resources, and apply to:&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;submitted jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that the user asked SLURM to execute, each of which may currently be either running or queued&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;running jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that are currently in execution&lt;br /&gt;
&lt;br /&gt;
Limits on submitted jobs are set via [[#SLURM Quality of Service (QOS)|QOSes]], while limits on running jobs are set via [[#research users and students users|user category]].&lt;br /&gt;
&lt;br /&gt;
The following table summarises what limits exist:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! &lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;running&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;submitted&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | global limits&amp;lt;br/&amp;gt;(system-wide)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to limits set by individual QOSes (below)&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | limits for&amp;lt;br/&amp;gt;individual&amp;lt;/br&amp;gt;QOSes&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to global limits (above)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for the&amp;#039;&amp;#039; &amp;lt;code&amp;gt;gpuwide&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;QOS&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for each of the other QOSes&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
= Job priority =&lt;br /&gt;
&lt;br /&gt;
Once the execution of a job has been requested, the job is not run immediately: it is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039; by SLURM, together with all the other jobs awaiting execution. The job on top of the queue at any time is the first to be put into execution as soon as the resources it requires are available. &lt;br /&gt;
&lt;br /&gt;
The order of the items in the job queue depends on their &amp;#039;&amp;#039;&amp;#039;priority&amp;#039;&amp;#039;&amp;#039;. The job with the highest priority is put by SLURM on top of the queue, while all other jobs are put in the queue in order of descending priority.&lt;br /&gt;
&lt;br /&gt;
The goal of SLURM is to maximise resource availability: i.e., to ensure the shorter possible wait time before job execution. To achieve this goal, in Mufasa SLURM is configured to &amp;#039;&amp;#039;&amp;#039;encourage users not to ask for resources or execution time that their job doesn&amp;#039;t need&amp;#039;&amp;#039;&amp;#039;. This is done via the priority mechanism: the more resources and/or time a job requests, the lower its priority will be; and the later it will be executed.&lt;br /&gt;
&lt;br /&gt;
Priority management in Mufasa is designed to set up a &amp;#039;&amp;#039;&amp;#039;virtuous cycle&amp;#039;&amp;#039;&amp;#039; where users, by carefully choosing what to ask for, obtain two results:&lt;br /&gt;
* they ensure that their job is executed as soon as possible;&lt;br /&gt;
* they leave as much as possible of Mufasa&amp;#039;s resources free for other users&amp;#039;s jobs.&lt;br /&gt;
&lt;br /&gt;
== Elements determining job priority ==&lt;br /&gt;
In Mufasa, the priority of a job is computed by SLURM according to the following elements:&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#research users and students users|User category]]&amp;#039;&amp;#039;&amp;#039; (i.e., &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;)&lt;br /&gt;
::: Used to provide higher priority to jobs run by &amp;#039;&amp;#039;&amp;#039;research personnel&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#SLURM Quality of Service (QOS)|QOS]]&amp;#039;&amp;#039;&amp;#039; used by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;less resources&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Number of CPUs&amp;#039;&amp;#039;&amp;#039; requested by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;a lower number of CPUs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job Age&amp;#039;&amp;#039;&amp;#039;, i.e. the time that the job has been waiting in the queue&lt;br /&gt;
::: Used to provide higher priority to jobs which have been &amp;#039;&amp;#039;&amp;#039;queued for a longer time&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;FairShare&amp;#039;&amp;#039;&amp;#039;, i.e. a factor computed by SLURM to balance use of the system by different users&lt;br /&gt;
::: Used to provide higher priority to jobs by users who &amp;#039;&amp;#039;&amp;#039;used Mufasa&amp;#039;s resources less than others&amp;#039;&amp;#039;&amp;#039; (see [[#How FairShare works|below]])&lt;br /&gt;
&lt;br /&gt;
You can understand how each of these elements contributes to the priority of the jobs waiting for execution (both yours and of other users) with command &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;. A tailored version of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; that only displays the parameters listed above is the following:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sprio -o &amp;quot;%6i %9u %.5Y %.10B %.5Q %.9T %.5A %.5F&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Important!&lt;br /&gt;
:: The output of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; is only relevant for jobs &amp;#039;&amp;#039;&amp;#039;not yet in execution&amp;#039;&amp;#039;&amp;#039;. For an already running job, the value shown by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; is the priority that the job would get if computed now, which usually does not correspond to the actual priority.&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Here is an example of the output of the command above:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
JOBID  USER      PRIOR      ASSOC   QOS      TRES   AGE FAIRS&lt;br /&gt;
5774   itombini   3492       2500   750  cpu=-800   906   136&lt;br /&gt;
5780   fbottazzi  2266        625   750  cpu=-600   420  1071&lt;br /&gt;
5783   mzoccola   1766        625   750  cpu=-800   217   974&lt;br /&gt;
5790   amoglia    2717       2500   375  cpu=-800   156   487&lt;br /&gt;
5791   qliu       2232       2500   375  cpu=-800   138    19&lt;br /&gt;
5792   fpastori   2197       2500   375  cpu=-800    84    39&lt;br /&gt;
5793   bagostini  2034        625  1500 cpu=-1200    18  1091&lt;br /&gt;
5794   lpassoni   2517       2500   750  cpu=-800     9    58&lt;br /&gt;
5795   lpassoni   2517       2500   750  cpu=-800     9    58&lt;br /&gt;
5796   adurat     3101       2500   750  cpu=-800     8   643&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
In it, column ASSOC (i.e., association) corresponds to user category, while column TRES corresponds to the number of CPUs. Note that the contribution of CPUs to priority is &amp;#039;&amp;#039;negative&amp;#039;&amp;#039;, i.e. it is a penalty. Column PRIOR provides the overall priority of each job.&lt;br /&gt;
&lt;br /&gt;
=== How FairShare works ===&lt;br /&gt;
In Mufasa 2.0, the FairShare of each user is computed according to these rules:&lt;br /&gt;
# It considers the quantity used by (current and past) user jobs of:&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;CPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of CPUs used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;RAM&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the amount of RAM used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;GPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of GPUs used&lt;br /&gt;
#** note: 40 GB GPUs have double the impact on FairShare than 20 GB GPUs&lt;br /&gt;
# The impact on FairShare of any use of resources is &amp;#039;&amp;#039;&amp;#039;proportional to its duration&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
#* example: using 32 GB of RAM for 48h has the same impact of using 64 GB of RAM for 24h&lt;br /&gt;
# FairShare has a &amp;quot;fading memory&amp;quot;: i.e., resource use has less and less impact on FairShare the farther in the past it is&lt;br /&gt;
#* consequence: over time, the &amp;quot;history&amp;quot; of a user gets forgotten by FairShare&lt;br /&gt;
&lt;br /&gt;
== How to maximise the priority of your jobs ==&lt;br /&gt;
&lt;br /&gt;
Every time you run a SLURM job, follow these guidelines:&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Choose the less powerful QOS compatible with the needs of your job&lt;br /&gt;
:: QOSes with access to less resources have higher base priority&lt;br /&gt;
&lt;br /&gt;
; Only request CPUs and RAM that your job really needs&lt;br /&gt;
:: Usually, code exploits multiple CPUs only if specifically designed to do so: if unsure, only ask for 1 CPU&lt;br /&gt;
:: The fewer resources your job asks for, the less it will wait before they become available&lt;br /&gt;
:: The more CPUs your job asks for, the stronger the penalty to its priority&lt;br /&gt;
:: Asking for fewer CPUs and/or less RAM improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Do not request more time than your jobs needs to complete&lt;br /&gt;
:: Make a worst-case estimate and only ask for that duration&lt;br /&gt;
:: Asking for less of Mufasa&amp;#039;s time improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Debug and test code using less powerful QOSes before running it with more powerful QOSes&lt;br /&gt;
:: Your test jobs will get a higher priority and your FairShare will improve&lt;br /&gt;
&lt;br /&gt;
; Cancel jobs that you don&amp;#039;t need them anymore&lt;br /&gt;
:: Always use [[User_Jobs#Cancelling_a_job_with_scancel|scancel]] to delete completed (or crashed) jobs: your Fairshare will improve&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Protip: it&amp;#039;s a good idea to [[User Jobs#Looking for unused GPUs|check for unused GPUs]] before choosing what to request. Requesting a GPU that is currently idle will help your job get executed sooner.&lt;br /&gt;
&lt;br /&gt;
= SLURM partitions =&lt;br /&gt;
&lt;br /&gt;
Partitions are another mechanism provided by SLURM to create different levels of access to system resources. Since in Mufasa 2.0 access to resources is controlled via [[#SLURM Quality of Service (QOS)|QOSes]], partitions are not very relevant. (This is one of the main differences between Mufasa 1.0 and Mufasa 2.0.)&lt;br /&gt;
&lt;br /&gt;
Note, however, that the default values for some features of SLURM jobs (e.g., duration) are [[#Default values|set by the partition]].&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, there is a single SLURM partition, called &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt;, and all jobs run on it. The state of &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt; can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sinfo -o &amp;quot;%10P %5a %9T %11L %10l&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PARTITION  AVAIL STATE     DEFAULTTIME TIMELIMIT &lt;br /&gt;
jobs*      up    idle      1:00:00     3-00:00:00&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where columns correspond to the following information:&lt;br /&gt;
&lt;br /&gt;
:; PARTITION&lt;br /&gt;
:: name of the partition; the asterisks indicates that it&amp;#039;s the default one&lt;br /&gt;
&lt;br /&gt;
:; AVAIL&lt;br /&gt;
:: state/availability of the partition: see [[#Partition availability|below]]&lt;br /&gt;
&lt;br /&gt;
:; STATE&lt;br /&gt;
:: state (using [https://slurm.schedmd.com/sinfo.html#SECTION_NODE-STATE-CODES these codes])&lt;br /&gt;
:: typical values are &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;mixed&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that some of the resources are busy executing jobs while other are idle, and &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;allocated&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that all of the resources are in use&lt;br /&gt;
&lt;br /&gt;
:; DEFAULTTIME&lt;br /&gt;
:: default runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:; TIMELIMIT&lt;br /&gt;
:: maximum runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
The asterisk at the end of the partition name indicates the default partition, i.e. the one on which jobs which do not ask for a specific partition are run.&lt;br /&gt;
&lt;br /&gt;
Command &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; does not tell you about the &amp;#039;&amp;#039;jobs&amp;#039;&amp;#039; submitted to a partition. This information is obtained, instead, with [[User Jobs#Inspecting jobs with squeue|command &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;]].&lt;br /&gt;
&lt;br /&gt;
== Partition availability ==&lt;br /&gt;
&lt;br /&gt;
The most important information that &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; provides is the &amp;#039;&amp;#039;&amp;#039;availability&amp;#039;&amp;#039;&amp;#039; (also called &amp;#039;&amp;#039;&amp;#039;state&amp;#039;&amp;#039;&amp;#039;) of partitions. This is shown in column &amp;quot;AVAIL&amp;quot;. Possible partition states are:&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is available&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed as soon as resources allow&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is in the process of becoming unavailable (i.e., of entering the &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; state: see below)&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is unavailable&lt;br /&gt;
:: There are no running jobs&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
When a partition goes from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; no harm is done to running jobs. In a normally functioning SLURM system, the passage from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; happens only when no jobs are running on the partition. If (e.g., due to a malfunction) the passage happens with jobs still running, they get killed.&lt;br /&gt;
&lt;br /&gt;
A partition in state &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; requires intervention by a [[Roles|Job Administrator]] to be restored to &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
== Default values ==&lt;br /&gt;
&lt;br /&gt;
The features of SLURM partitions, including the &amp;#039;&amp;#039;&amp;#039;default values&amp;#039;&amp;#039;&amp;#039; which are applied to jobs that do not make explicit requests, can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scontrol show partition&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PartitionName=jobs&lt;br /&gt;
   AllowGroups=ALL AllowAccounts=ALL AllowQos=nogpu,gpulight,gpu,gpuwide,gpuheavy-20,gpuheavy-40&lt;br /&gt;
   AllocNodes=ALL Default=YES QoS=N/A&lt;br /&gt;
=&amp;gt; DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO&lt;br /&gt;
   MaxNodes=UNLIMITED MaxTime=3-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED&lt;br /&gt;
   Nodes=gn01&lt;br /&gt;
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO&lt;br /&gt;
   OverTimeLimit=NONE PreemptMode=OFF&lt;br /&gt;
   State=UP TotalCPUs=48 TotalNodes=1 SelectTypeParameters=NONE&lt;br /&gt;
   JobDefaults=(null)&lt;br /&gt;
=&amp;gt; DefMemPerNode=4096 MaxMemPerNode=UNLIMITED&lt;br /&gt;
   TRES=cpu=48,mem=1011435M,node=1,billing=49,gres/gpu=13,gres/gpu:3g.20gb=5,gres/gpu:40gb=3,gres/gpu:4g.20gb=5&lt;br /&gt;
   TRESBillingWeights=cpu=1.0,gres/gpu:3g.20gb=6.0,gres/gpu:4g.20gb=6.0,gres/gpu:40gb=6.0,mem=0.05g&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In the example, we have highlighted with  &amp;#039;&amp;#039;&amp;#039;=&amp;gt;&amp;#039;&amp;#039;&amp;#039;  the most relevant default values for Mufasa users, i.e.:&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefaultTime&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default execution time assigned to a job run on the partition (e.g., 1 hour)&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefMemPerNode&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default amount of RAM assigned to a job run on the partition (e.g., 4GB)&lt;br /&gt;
&lt;br /&gt;
= System resources subjected to limitations =&lt;br /&gt;
&lt;br /&gt;
In systems based on SLURM like Mufasa, &amp;#039;&amp;#039;&amp;#039;TRES (Trackable RESources)&amp;#039;&amp;#039;&amp;#039; are (from [https://slurm.schedmd.com/tres.html SLURM&amp;#039;s documentation] &amp;quot;&amp;#039;&amp;#039;resources that can be tracked for usage or used to enforce limits against.&amp;#039;&amp;#039;&amp;quot;&lt;br /&gt;
&lt;br /&gt;
TRES include CPUs, RAM and &amp;#039;&amp;#039;&amp;#039;GRES&amp;#039;&amp;#039;&amp;#039;. The last term stands for &amp;#039;&amp;#039;Generic RESources&amp;#039;&amp;#039; that a job may need for its execution. In Mufasa, the only &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; resources are the GPUs.&lt;br /&gt;
&lt;br /&gt;
== &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax ==&lt;br /&gt;
&lt;br /&gt;
To ask SLURM to assign GRES resources (i.e., GPUs) to a job, a special syntax must be used. Precisely, the name of each GPU resource takes the form&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:Name:Type&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
Considering the [[System#CPUs and GPUs|GPU complement of Mufasa]], &amp;lt;code&amp;gt;Type&amp;lt;/code&amp;gt; takes the following values:&lt;br /&gt;
&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:40gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 40 Gbytes of RAM&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:4g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 4 compute units&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 3 compute units&lt;br /&gt;
&lt;br /&gt;
So, for instance,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
identifies a resource corresponding to a GPU with 20 GB of RAM and 3 compute units.&lt;br /&gt;
&lt;br /&gt;
When asking for a GRES resource (e.g., in an &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command or an &amp;lt;code&amp;gt;SBATCH&amp;lt;/code&amp;gt; directive of an [[User Jobs#Using execution scripts to run jobs|execution script]]), the syntax required by SLURM is&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:&amp;lt;Type&amp;gt;:&amp;lt;Quantity&amp;gt;&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
where &amp;lt;code&amp;gt;Quantity&amp;lt;/code&amp;gt; is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of type &amp;lt;code&amp;gt;4g.20gb&amp;lt;/code&amp;gt; the syntax is&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:4g.20gb:2&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SLURM&amp;#039;s &amp;#039;&amp;#039;generic resources&amp;#039;&amp;#039; are defined in &amp;lt;code&amp;gt;/etc/slurm/gres.conf&amp;lt;/code&amp;gt;. In order to make GPUs available to SLURM&amp;#039;s &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; management, Mufasa makes use of Nvidia&amp;#039;s [https://developer.nvidia.com/nvidia-management-library-nvml NVML library]. For additional information see [https://slurm.schedmd.com/gres.html SLURM&amp;#039;s documentation].&lt;/div&gt;</summary>
		<author><name>GiulioFontana</name></author>
	</entry>
	<entry>
		<id>https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2581</id>
		<title>SLURM</title>
		<link rel="alternate" type="text/html" href="https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2581"/>
		<updated>2026-11-03T14:45:48Z</updated>

		<summary type="html">&lt;p&gt;GiulioFontana: /* Elements determining job priority */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page presents the features of SLURM that are most relevant to Mufasa&amp;#039;s [[Roles|Job Users]]. Job Users can submit jobs for execution, cancel their own jobs, and see other users&amp;#039; jobs (but not intervene on them).&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa &amp;#039;&amp;#039;&amp;#039;must use SLURM&amp;#039;&amp;#039;&amp;#039; to run resource-heavy processes, i.e. computing jobs that require one or more of the following:&lt;br /&gt;
* GPUs&lt;br /&gt;
* multiple CPUs&lt;br /&gt;
* powerful CPUs&lt;br /&gt;
* a significant amount of RAM&lt;br /&gt;
&lt;br /&gt;
In fact, only processes run via SLURM have access to all the resources of Mufasa. Processes run outside SLURM are executed by the [[System#Login server|login server]] virtual machine, which has minimal resources and no GPUs. Using SLURM is therefore the only way to execute resource-heavy jobs on Mufasa (this is a key difference between Mufasa 1.0 and Mufasa 2.0).&lt;br /&gt;
&lt;br /&gt;
= SLURM in a nutshell =&lt;br /&gt;
&lt;br /&gt;
Computation jobs on Mufasa needs to be launched via [[System#The SLURM job scheduling system|SLURM]]. SLURM provides jobs with access to the [[#System resources subjected to limitations|physical resources]] of Mufasa, such as CPUs, GPUs and RAM. Thanks to SLURM, processing jobs share system resources, optimising their occupation and availability. &lt;br /&gt;
&lt;br /&gt;
When a user runs a job, the job does not get executed immediately and is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039;. SLURM executes jobs according to their order in the queue: the top job in the queue gets executed as soon as the necessary resources are available, while jobs lower in the queue wait longer. The position of a job in the queue is due to the &amp;#039;&amp;#039;&amp;#039;[[#Job priority|priority]]&amp;#039;&amp;#039;&amp;#039; assigned to it by SLURM, with higher-priority jobs closer to the top. As a general rule:&lt;br /&gt;
&lt;br /&gt;
;: &amp;#039;&amp;#039;&amp;#039;the greater the fraction of Mufasa&amp;#039;s overall resources that a job asks for, the lower the job&amp;#039;s priority will be&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
The priority mechanism is used to encourage users to use Mufasa&amp;#039;s resources (i.e.: GPUs, CPUs, RAM, execution time) in an effective and equitable manner. This page includes a [[#How_to_maximise_the_priority_of_your_jobs|chart explaining how to maximise the priority of your jobs]].&lt;br /&gt;
&lt;br /&gt;
The &amp;#039;&amp;#039;&amp;#039;time&amp;#039;&amp;#039;&amp;#039; available to a job for its execution is controlled by SLURM. When a user requests execution of a job, they must specify the duration of the time slot that the job needs. The job must complete its execution before the end of the requested time slot, because at the end of such slot it gets killed by SLURM.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0 access to system resources is managed via SLURM&amp;#039;s [[#SLURM Quality of Service (QOS)|Quality of Service (QOS)]] mechanism (Mufasa 1.0 used [[#SLURM_partitions|partitions]] instead). While [[User Jobs#Running jobs with SLURM|launching a processing job via SLURM]], the user must always specify the chosen QOS. QOSes differ in the set of resources that they provide access to because each of them is designed to fit a given type of job.&lt;br /&gt;
&lt;br /&gt;
Mufasa sets limits to the number of jobs by the same user. This page includes a [[#Limits on jobs by the same user|table summarising such limits]].&lt;br /&gt;
&lt;br /&gt;
= SLURM Quality of Service (QOS) =&lt;br /&gt;
&lt;br /&gt;
Through &amp;#039;&amp;#039;&amp;#039;Quality of Services&amp;#039;&amp;#039;&amp;#039; (&amp;#039;&amp;#039;&amp;#039;QOSes&amp;#039;&amp;#039;&amp;#039;), SLURM lets system configurators assign a name to a set of related constraints.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, QOSes are used to define different levels of access to the server&amp;#039;s resources. When [[User Jobs|executing a job with SLURM]], a user must always &amp;#039;&amp;#039;&amp;#039;specify the QOS&amp;#039;&amp;#039;&amp;#039; that their job will use: this choice, in turn, determines what resources the job is able to access and influences the [[#Job priority|priority]] of the job.&lt;br /&gt;
&lt;br /&gt;
Mufasa&amp;#039;s QOSes and their features can be inspected with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list qos format=name%-11,priority,MaxSubmit,maxwall,maxtres%-80,mintres%-18&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following (see below for the same data in table form):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
Name          Priority MaxSubmit     MaxWall MaxTRES                                                                          MinTRES            &lt;br /&gt;
----------- ---------- --------- ----------- -------------------------------------------------------------------------------- ------------------ &lt;br /&gt;
normal               0                                                                                                                           &lt;br /&gt;
nogpu                4         1  3-00:00:00 cpu=16,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=128G                    &lt;br /&gt;
gpuheavy-20          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=2,mem=128G             gres/gpu:4g.20gb=1 &lt;br /&gt;
gpuheavy-40          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=1,gres/gpu:4g.20gb=0,mem=128G             gres/gpu:40gb=1    &lt;br /&gt;
gpulight             8         1    12:00:00 cpu=2,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpu                  2         1  1-00:00:00 cpu=8,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpuwide              2         2  1-00:00:00 cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=1,mem=64G              gres/gpu:4g.20gb=1 &lt;br /&gt;
build               32         1    02:00:00 cpu=2,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=16G&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The columns of this output are the following:&lt;br /&gt;
&lt;br /&gt;
:; Name&lt;br /&gt;
:: name of the QOS&lt;br /&gt;
&lt;br /&gt;
:; Priority&lt;br /&gt;
:: priority tier associated to the QOS (higher value = higher priority): see [[#Job priority|Job priority]] for details&lt;br /&gt;
&lt;br /&gt;
:; MaxSubmit&lt;br /&gt;
:: maximum number of jobs from a single user that can be submitted to SLURM with this QOS; submitted jobs include both running and queued jobs&lt;br /&gt;
:: See [[#Limits on jobs by the same user|Limits on jobs by the same user]] for an overview of the limits on jobs set by Mufasa.&lt;br /&gt;
&lt;br /&gt;
:; MaxWall&lt;br /&gt;
:: maximum wall clock duration of the jobs using the QOS (after which they are killed by SLURM), in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
:: For some QOSes these are not set: it means that they are determined by the [[#SLURM partitions|partition]]. Partitions also define the [[#Default values|default duration]] of jobs.&lt;br /&gt;
&lt;br /&gt;
:; MaxTRES&lt;br /&gt;
:: amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) available to a job using the QOS, where&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;cpu=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of CPUs (i.e., processor cores) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of CPUs specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;gres/&amp;#039;&amp;#039;gpu:Type&amp;#039;&amp;#039;=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of GPUs of class &amp;lt;code&amp;gt;&amp;#039;&amp;#039;Type&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; (see [[User Jobs#gres syntax|&amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax]]) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; (for QOSes that allow access to GPUs) if not specified, the job cannot be launched&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;mem=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;G&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum amount of system RAM is &amp;#039;&amp;#039;K&amp;#039;&amp;#039; GBytes&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of RAM specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
&lt;br /&gt;
:; MinTRES&lt;br /&gt;
:: minimum amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) that a job using the QOS must request in order to actually get executed by SLURM.&lt;br /&gt;
:: (If your job does not actually need these resources, you&amp;#039;ve chosen the wrong QOS: you can use one with a higher priority.)&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; QOS is the default one, and exists only to ensure that users always specify a QOS when running a job. Since &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; has zero priority and no resources, a job run using this QOS would never be run.&lt;br /&gt;
&lt;br /&gt;
The information provided by the &amp;lt;code&amp;gt;sacctmgr list qos&amp;lt;/code&amp;gt; command above is summarised by the following table:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! QOS&lt;br /&gt;
! Priority&amp;lt;/br&amp;gt;tier&lt;br /&gt;
! Max&amp;lt;/br&amp;gt;Submit&lt;br /&gt;
! MaxWall&amp;lt;/br&amp;gt;[h]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;CPUs&lt;br /&gt;
! max RAM&amp;lt;/br&amp;gt;[GB]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;3g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;4g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;40GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | build&lt;br /&gt;
| 32&lt;br /&gt;
| 1&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 16&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpulight&lt;br /&gt;
| 8&lt;br /&gt;
| 1&lt;br /&gt;
| 12&lt;br /&gt;
| 2&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | nogpu&lt;br /&gt;
| 4&lt;br /&gt;
| 1&lt;br /&gt;
| 72&lt;br /&gt;
| 16&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpu&lt;br /&gt;
| 2&lt;br /&gt;
| 1&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuwide&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-20&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| 2 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-40&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
A key piece of information in the table above is the &amp;#039;&amp;#039;&amp;#039;priority tier&amp;#039;&amp;#039;&amp;#039; associated to each QOS. In Mufasa 2.0, priority tiers are used to encourage users to use the &amp;#039;&amp;#039;&amp;#039;least powerful QOS that is compatible with their needs&amp;#039;&amp;#039;&amp;#039;, where &amp;quot;powerful&amp;quot; means &amp;quot;rich with resources&amp;quot;. Less powerful QOSes increase the priority of the jobs that use them, so these jobs tend to be executed sooner. See [[#Job priority|Job priority]] for details about how priority affects the execution order of jobs in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
== The &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS ==&lt;br /&gt;
&lt;br /&gt;
This QOS is specifically designed to be used by Mufasa users to &amp;#039;&amp;#039;&amp;#039;quickly build [[System#Containers|container images]]&amp;#039;&amp;#039;&amp;#039;. Its associated priority tier is very high: this, combined with the fact that jobs using this QOS require few resources, means that such jobs usually get executed very soon.&lt;br /&gt;
&lt;br /&gt;
On the other hand, the limited resources and the lack of access to the GPUs make the &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS unsuitable for other tasks.&lt;br /&gt;
&lt;br /&gt;
See [[Singularity#Building Singularity images|Building Singularity images]] for directions about building Singularity container images.&lt;br /&gt;
&lt;br /&gt;
== Restricted QOSes ==&lt;br /&gt;
&lt;br /&gt;
In Mufasa, the most powerful QOSes are reserved to researchers (called &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users: these include academic personnel and Ph.D. students). M.Sc. students, called &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users, cannot use them while running jobs.&lt;br /&gt;
&lt;br /&gt;
See [[#research users and students users|below]] to understand how &amp;#039;&amp;#039;user categories&amp;#039;&amp;#039; work in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
= &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users =&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa belong to two &amp;#039;&amp;#039;&amp;#039;user categories&amp;#039;&amp;#039;&amp;#039;. Categories provide users with different access to  to Mufasa&amp;#039;s resources: the idea being to provide researchers with more access while still letting students benefit from the server&amp;#039;s capabilities.&lt;br /&gt;
&lt;br /&gt;
User categories in Mufasa are the following:&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. academic personnel and Ph.D. students&lt;br /&gt;
::: * can use all QOSes, including the the [[#Restricted QOSes|restricted ones]]&lt;br /&gt;
::: * their jobs have a higher &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a higher number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. M.Sc. students&lt;br /&gt;
::: * cannot use the [[#Restricted QOSes|restricted QOSes]]&lt;br /&gt;
::: * their jobs have a lower &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a lower number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
You can inspect the differences in priority and running jobs between &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association format=&amp;quot;account,priority,maxjobs&amp;quot; | grep -E &amp;#039;Account|research|students&amp;#039;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
   Account   Priority MaxJobs &lt;br /&gt;
  research          4       2 &lt;br /&gt;
  students          1       1&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
To know what limits apply to your own user, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association where user=$USER format=&amp;quot;user,priority,maxjobs,qos%-60&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
      User   Priority MaxJobs QOS                                                          &lt;br /&gt;
---------- ---------- ------- ------------------------------------------------------------ &lt;br /&gt;
    preali          4       2 build,gpu,gpuheavy-20,gpuheavy-40,gpulight,gpuwide,nogpu&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The list under &amp;quot;QOS&amp;quot; shows what QOSes your user is allowed to use when [[User Jobs|running jobs]]. &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users can use all of them, while &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users can only access a subset of them.&lt;br /&gt;
&lt;br /&gt;
= Limits on jobs by the same user =&lt;br /&gt;
&lt;br /&gt;
Mufasa enforces limits on the number of jobs from a single user. Such limits aim at preventing users from &amp;quot;hogging&amp;quot; system resources, and apply to:&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;submitted jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that the user asked SLURM to execute, each of which may currently be either running or queued&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;running jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that are currently in execution&lt;br /&gt;
&lt;br /&gt;
Limits on submitted jobs are set via [[#SLURM Quality of Service (QOS)|QOSes]], while limits on running jobs are set via [[#research users and students users|user category]].&lt;br /&gt;
&lt;br /&gt;
The following table summarises what limits exist:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! &lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;running&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;submitted&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | global limits&amp;lt;br/&amp;gt;(system-wide)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to limits set by individual QOSes (below)&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | limits for&amp;lt;br/&amp;gt;individual&amp;lt;/br&amp;gt;QOSes&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to global limits (above)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for the&amp;#039;&amp;#039; &amp;lt;code&amp;gt;gpuwide&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;QOS&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for each of the other QOSes&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
= Job priority =&lt;br /&gt;
&lt;br /&gt;
Once the execution of a job has been requested, the job is not run immediately: it is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039; by SLURM, together with all the other jobs awaiting execution. The job on top of the queue at any time is the first to be put into execution as soon as the resources it requires are available. &lt;br /&gt;
&lt;br /&gt;
The order of the items in the job queue depends on their &amp;#039;&amp;#039;&amp;#039;priority&amp;#039;&amp;#039;&amp;#039;. The job with the highest priority is put by SLURM on top of the queue, while all other jobs are put in the queue in order of descending priority.&lt;br /&gt;
&lt;br /&gt;
The goal of SLURM is to maximise resource availability: i.e., to ensure the shorter possible wait time before job execution. To achieve this goal, in Mufasa SLURM is configured to &amp;#039;&amp;#039;&amp;#039;encourage users not to ask for resources or execution time that their job doesn&amp;#039;t need&amp;#039;&amp;#039;&amp;#039;. This is done via the priority mechanism: the more resources and/or time a job requests, the lower its priority will be; and the later it will be executed.&lt;br /&gt;
&lt;br /&gt;
Priority management in Mufasa is designed to set up a &amp;#039;&amp;#039;&amp;#039;virtuous cycle&amp;#039;&amp;#039;&amp;#039; where users, by carefully choosing what to ask for, obtain two results:&lt;br /&gt;
* they ensure that their job is executed as soon as possible;&lt;br /&gt;
* they leave as much as possible of Mufasa&amp;#039;s resources free for other users&amp;#039;s jobs.&lt;br /&gt;
&lt;br /&gt;
== Elements determining job priority ==&lt;br /&gt;
In Mufasa, the priority of a job is computed by SLURM according to the following elements:&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#research users and students users|User category]]&amp;#039;&amp;#039;&amp;#039; (i.e., &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;)&lt;br /&gt;
::: Used to provide higher priority to jobs run by &amp;#039;&amp;#039;&amp;#039;research personnel&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#SLURM Quality of Service (QOS)|QOS]]&amp;#039;&amp;#039;&amp;#039; used by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;less resources&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Number of CPUs&amp;#039;&amp;#039;&amp;#039; requested by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;a lower number of CPUs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job Age&amp;#039;&amp;#039;&amp;#039;, i.e. the time that the job has been waiting in the queue&lt;br /&gt;
::: Used to provide higher priority to jobs which have been &amp;#039;&amp;#039;&amp;#039;queued for a longer time&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;FairShare&amp;#039;&amp;#039;&amp;#039;, i.e. a factor computed by SLURM to balance use of the system by different users&lt;br /&gt;
::: Used to provide higher priority to jobs by users who &amp;#039;&amp;#039;&amp;#039;used Mufasa&amp;#039;s resources less than others&amp;#039;&amp;#039;&amp;#039; (see [[#How FairShare works|below]])&lt;br /&gt;
&lt;br /&gt;
You can understand how each of these elements contributes to the priority of the jobs waiting for execution (both yours and of other users) with command &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;. A tailored version of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; that only displays the parameters listed above is the following:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sprio -o &amp;quot;%6i %9u %.5Y %.10B %.5Q %.9T %.5A %.5F&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Important!&lt;br /&gt;
:: The output of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; is only relevant for jobs &amp;#039;&amp;#039;&amp;#039;not yet in execution&amp;#039;&amp;#039;&amp;#039;. For an already running job, the value shown by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; is the priority that the job would get if computed now, which usually does not correspond to the actual priority.&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Here is an example of the output of the command above:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
JOBID  USER      PRIOR      ASSOC   QOS      TRES   AGE FAIRS&lt;br /&gt;
5774   itombini   3492       2500   750  cpu=-800   906   136&lt;br /&gt;
5780   fbottazzi  2266        625   750  cpu=-600   420  1071&lt;br /&gt;
5783   mzoccola   1766        625   750  cpu=-800   217   974&lt;br /&gt;
5790   amoglia    2717       2500   375  cpu=-800   156   487&lt;br /&gt;
5791   qliu       2232       2500   375  cpu=-800   138    19&lt;br /&gt;
5792   fpastori   2197       2500   375  cpu=-800    84    39&lt;br /&gt;
5793   bagostini  2034        625  1500 cpu=-1200    18  1091&lt;br /&gt;
5794   lpassoni   2517       2500   750  cpu=-800     9    58&lt;br /&gt;
5795   lpassoni   2517       2500   750  cpu=-800     9    58&lt;br /&gt;
5796   adurat     3101       2500   750  cpu=-800     8   643&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
In it, column ASSOC (i.e., association) corresponds to user category, while column TRES corresponds to the number of CPUs. Note that the contribution of CPUs to priority is &amp;#039;&amp;#039;negative&amp;#039;&amp;#039;, i.e. it is a penalty. Column PRIOR provides the overall priority of each job.&lt;br /&gt;
&lt;br /&gt;
=== How FairShare works ===&lt;br /&gt;
In Mufasa 2.0, the FairShare of each user is computed according to these rules:&lt;br /&gt;
# It considers the quantity used by (current and past) user jobs of:&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;CPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of CPUs used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;RAM&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the amount of RAM used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;GPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of GPUs used&lt;br /&gt;
#** note: 40 GB GPUs have double the impact on FairShare than 20 GB GPUs&lt;br /&gt;
# The impact on FairShare of any use of resources is &amp;#039;&amp;#039;&amp;#039;proportional to its duration&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
#* example: using 32 GB of RAM for 48h has the same impact of using 64 GB of RAM for 24h&lt;br /&gt;
# FairShare has a &amp;quot;fading memory&amp;quot;: i.e., resource use has less and less impact on FairShare the farther in the past it is&lt;br /&gt;
#* consequence: over time, the &amp;quot;history&amp;quot; of a user gets forgotten by FairShare&lt;br /&gt;
&lt;br /&gt;
== How to maximise the priority of your jobs ==&lt;br /&gt;
&lt;br /&gt;
Every time you run a SLURM job, follow these guidelines:&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Choose the less powerful QOS compatible with the needs of your job&lt;br /&gt;
:: QOSes with access to less resources have higher base priority&lt;br /&gt;
&lt;br /&gt;
; Only request CPUs and RAM that your job really needs&lt;br /&gt;
:: Usually, code exploits multiple CPUs only if specifically designed to do so: if unsure, only ask for 1 CPU&lt;br /&gt;
:: The fewer resources your job asks for, the less it will wait before they become available&lt;br /&gt;
:: Asking for fewer CPUs and/or less RAM improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Do not request more time than your jobs needs to complete&lt;br /&gt;
:: Make a worst-case estimate and only ask for that duration&lt;br /&gt;
:: Asking for less of Mufasa&amp;#039;s time improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Debug and test code using less powerful QOSes before running it with more powerful QOSes&lt;br /&gt;
:: Your test jobs will get a higher priority and your FairShare will improve&lt;br /&gt;
&lt;br /&gt;
; Cancel jobs that you don&amp;#039;t need them anymore&lt;br /&gt;
:: Always use [[User_Jobs#Cancelling_a_job_with_scancel|scancel]] to delete completed (or crashed) jobs: your Fairshare will improve&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Protip: it&amp;#039;s a good idea to [[User Jobs#Looking for unused GPUs|check for unused GPUs]] before choosing what to request. Requesting a GPU that is currently idle will help your job get executed sooner.&lt;br /&gt;
&lt;br /&gt;
= SLURM partitions =&lt;br /&gt;
&lt;br /&gt;
Partitions are another mechanism provided by SLURM to create different levels of access to system resources. Since in Mufasa 2.0 access to resources is controlled via [[#SLURM Quality of Service (QOS)|QOSes]], partitions are not very relevant. (This is one of the main differences between Mufasa 1.0 and Mufasa 2.0.)&lt;br /&gt;
&lt;br /&gt;
Note, however, that the default values for some features of SLURM jobs (e.g., duration) are [[#Default values|set by the partition]].&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, there is a single SLURM partition, called &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt;, and all jobs run on it. The state of &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt; can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sinfo -o &amp;quot;%10P %5a %9T %11L %10l&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PARTITION  AVAIL STATE     DEFAULTTIME TIMELIMIT &lt;br /&gt;
jobs*      up    idle      1:00:00     3-00:00:00&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where columns correspond to the following information:&lt;br /&gt;
&lt;br /&gt;
:; PARTITION&lt;br /&gt;
:: name of the partition; the asterisks indicates that it&amp;#039;s the default one&lt;br /&gt;
&lt;br /&gt;
:; AVAIL&lt;br /&gt;
:: state/availability of the partition: see [[#Partition availability|below]]&lt;br /&gt;
&lt;br /&gt;
:; STATE&lt;br /&gt;
:: state (using [https://slurm.schedmd.com/sinfo.html#SECTION_NODE-STATE-CODES these codes])&lt;br /&gt;
:: typical values are &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;mixed&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that some of the resources are busy executing jobs while other are idle, and &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;allocated&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that all of the resources are in use&lt;br /&gt;
&lt;br /&gt;
:; DEFAULTTIME&lt;br /&gt;
:: default runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:; TIMELIMIT&lt;br /&gt;
:: maximum runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
The asterisk at the end of the partition name indicates the default partition, i.e. the one on which jobs which do not ask for a specific partition are run.&lt;br /&gt;
&lt;br /&gt;
Command &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; does not tell you about the &amp;#039;&amp;#039;jobs&amp;#039;&amp;#039; submitted to a partition. This information is obtained, instead, with [[User Jobs#Inspecting jobs with squeue|command &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;]].&lt;br /&gt;
&lt;br /&gt;
== Partition availability ==&lt;br /&gt;
&lt;br /&gt;
The most important information that &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; provides is the &amp;#039;&amp;#039;&amp;#039;availability&amp;#039;&amp;#039;&amp;#039; (also called &amp;#039;&amp;#039;&amp;#039;state&amp;#039;&amp;#039;&amp;#039;) of partitions. This is shown in column &amp;quot;AVAIL&amp;quot;. Possible partition states are:&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is available&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed as soon as resources allow&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is in the process of becoming unavailable (i.e., of entering the &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; state: see below)&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is unavailable&lt;br /&gt;
:: There are no running jobs&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
When a partition goes from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; no harm is done to running jobs. In a normally functioning SLURM system, the passage from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; happens only when no jobs are running on the partition. If (e.g., due to a malfunction) the passage happens with jobs still running, they get killed.&lt;br /&gt;
&lt;br /&gt;
A partition in state &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; requires intervention by a [[Roles|Job Administrator]] to be restored to &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
== Default values ==&lt;br /&gt;
&lt;br /&gt;
The features of SLURM partitions, including the &amp;#039;&amp;#039;&amp;#039;default values&amp;#039;&amp;#039;&amp;#039; which are applied to jobs that do not make explicit requests, can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scontrol show partition&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PartitionName=jobs&lt;br /&gt;
   AllowGroups=ALL AllowAccounts=ALL AllowQos=nogpu,gpulight,gpu,gpuwide,gpuheavy-20,gpuheavy-40&lt;br /&gt;
   AllocNodes=ALL Default=YES QoS=N/A&lt;br /&gt;
=&amp;gt; DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO&lt;br /&gt;
   MaxNodes=UNLIMITED MaxTime=3-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED&lt;br /&gt;
   Nodes=gn01&lt;br /&gt;
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO&lt;br /&gt;
   OverTimeLimit=NONE PreemptMode=OFF&lt;br /&gt;
   State=UP TotalCPUs=48 TotalNodes=1 SelectTypeParameters=NONE&lt;br /&gt;
   JobDefaults=(null)&lt;br /&gt;
=&amp;gt; DefMemPerNode=4096 MaxMemPerNode=UNLIMITED&lt;br /&gt;
   TRES=cpu=48,mem=1011435M,node=1,billing=49,gres/gpu=13,gres/gpu:3g.20gb=5,gres/gpu:40gb=3,gres/gpu:4g.20gb=5&lt;br /&gt;
   TRESBillingWeights=cpu=1.0,gres/gpu:3g.20gb=6.0,gres/gpu:4g.20gb=6.0,gres/gpu:40gb=6.0,mem=0.05g&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In the example, we have highlighted with  &amp;#039;&amp;#039;&amp;#039;=&amp;gt;&amp;#039;&amp;#039;&amp;#039;  the most relevant default values for Mufasa users, i.e.:&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefaultTime&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default execution time assigned to a job run on the partition (e.g., 1 hour)&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefMemPerNode&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default amount of RAM assigned to a job run on the partition (e.g., 4GB)&lt;br /&gt;
&lt;br /&gt;
= System resources subjected to limitations =&lt;br /&gt;
&lt;br /&gt;
In systems based on SLURM like Mufasa, &amp;#039;&amp;#039;&amp;#039;TRES (Trackable RESources)&amp;#039;&amp;#039;&amp;#039; are (from [https://slurm.schedmd.com/tres.html SLURM&amp;#039;s documentation] &amp;quot;&amp;#039;&amp;#039;resources that can be tracked for usage or used to enforce limits against.&amp;#039;&amp;#039;&amp;quot;&lt;br /&gt;
&lt;br /&gt;
TRES include CPUs, RAM and &amp;#039;&amp;#039;&amp;#039;GRES&amp;#039;&amp;#039;&amp;#039;. The last term stands for &amp;#039;&amp;#039;Generic RESources&amp;#039;&amp;#039; that a job may need for its execution. In Mufasa, the only &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; resources are the GPUs.&lt;br /&gt;
&lt;br /&gt;
== &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax ==&lt;br /&gt;
&lt;br /&gt;
To ask SLURM to assign GRES resources (i.e., GPUs) to a job, a special syntax must be used. Precisely, the name of each GPU resource takes the form&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:Name:Type&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
Considering the [[System#CPUs and GPUs|GPU complement of Mufasa]], &amp;lt;code&amp;gt;Type&amp;lt;/code&amp;gt; takes the following values:&lt;br /&gt;
&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:40gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 40 Gbytes of RAM&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:4g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 4 compute units&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 3 compute units&lt;br /&gt;
&lt;br /&gt;
So, for instance,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
identifies a resource corresponding to a GPU with 20 GB of RAM and 3 compute units.&lt;br /&gt;
&lt;br /&gt;
When asking for a GRES resource (e.g., in an &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command or an &amp;lt;code&amp;gt;SBATCH&amp;lt;/code&amp;gt; directive of an [[User Jobs#Using execution scripts to run jobs|execution script]]), the syntax required by SLURM is&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:&amp;lt;Type&amp;gt;:&amp;lt;Quantity&amp;gt;&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
where &amp;lt;code&amp;gt;Quantity&amp;lt;/code&amp;gt; is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of type &amp;lt;code&amp;gt;4g.20gb&amp;lt;/code&amp;gt; the syntax is&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:4g.20gb:2&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SLURM&amp;#039;s &amp;#039;&amp;#039;generic resources&amp;#039;&amp;#039; are defined in &amp;lt;code&amp;gt;/etc/slurm/gres.conf&amp;lt;/code&amp;gt;. In order to make GPUs available to SLURM&amp;#039;s &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; management, Mufasa makes use of Nvidia&amp;#039;s [https://developer.nvidia.com/nvidia-management-library-nvml NVML library]. For additional information see [https://slurm.schedmd.com/gres.html SLURM&amp;#039;s documentation].&lt;/div&gt;</summary>
		<author><name>GiulioFontana</name></author>
	</entry>
	<entry>
		<id>https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2580</id>
		<title>SLURM</title>
		<link rel="alternate" type="text/html" href="https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2580"/>
		<updated>2026-11-03T14:44:10Z</updated>

		<summary type="html">&lt;p&gt;GiulioFontana: /* Elements determining job priority */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page presents the features of SLURM that are most relevant to Mufasa&amp;#039;s [[Roles|Job Users]]. Job Users can submit jobs for execution, cancel their own jobs, and see other users&amp;#039; jobs (but not intervene on them).&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa &amp;#039;&amp;#039;&amp;#039;must use SLURM&amp;#039;&amp;#039;&amp;#039; to run resource-heavy processes, i.e. computing jobs that require one or more of the following:&lt;br /&gt;
* GPUs&lt;br /&gt;
* multiple CPUs&lt;br /&gt;
* powerful CPUs&lt;br /&gt;
* a significant amount of RAM&lt;br /&gt;
&lt;br /&gt;
In fact, only processes run via SLURM have access to all the resources of Mufasa. Processes run outside SLURM are executed by the [[System#Login server|login server]] virtual machine, which has minimal resources and no GPUs. Using SLURM is therefore the only way to execute resource-heavy jobs on Mufasa (this is a key difference between Mufasa 1.0 and Mufasa 2.0).&lt;br /&gt;
&lt;br /&gt;
= SLURM in a nutshell =&lt;br /&gt;
&lt;br /&gt;
Computation jobs on Mufasa needs to be launched via [[System#The SLURM job scheduling system|SLURM]]. SLURM provides jobs with access to the [[#System resources subjected to limitations|physical resources]] of Mufasa, such as CPUs, GPUs and RAM. Thanks to SLURM, processing jobs share system resources, optimising their occupation and availability. &lt;br /&gt;
&lt;br /&gt;
When a user runs a job, the job does not get executed immediately and is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039;. SLURM executes jobs according to their order in the queue: the top job in the queue gets executed as soon as the necessary resources are available, while jobs lower in the queue wait longer. The position of a job in the queue is due to the &amp;#039;&amp;#039;&amp;#039;[[#Job priority|priority]]&amp;#039;&amp;#039;&amp;#039; assigned to it by SLURM, with higher-priority jobs closer to the top. As a general rule:&lt;br /&gt;
&lt;br /&gt;
;: &amp;#039;&amp;#039;&amp;#039;the greater the fraction of Mufasa&amp;#039;s overall resources that a job asks for, the lower the job&amp;#039;s priority will be&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
The priority mechanism is used to encourage users to use Mufasa&amp;#039;s resources (i.e.: GPUs, CPUs, RAM, execution time) in an effective and equitable manner. This page includes a [[#How_to_maximise_the_priority_of_your_jobs|chart explaining how to maximise the priority of your jobs]].&lt;br /&gt;
&lt;br /&gt;
The &amp;#039;&amp;#039;&amp;#039;time&amp;#039;&amp;#039;&amp;#039; available to a job for its execution is controlled by SLURM. When a user requests execution of a job, they must specify the duration of the time slot that the job needs. The job must complete its execution before the end of the requested time slot, because at the end of such slot it gets killed by SLURM.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0 access to system resources is managed via SLURM&amp;#039;s [[#SLURM Quality of Service (QOS)|Quality of Service (QOS)]] mechanism (Mufasa 1.0 used [[#SLURM_partitions|partitions]] instead). While [[User Jobs#Running jobs with SLURM|launching a processing job via SLURM]], the user must always specify the chosen QOS. QOSes differ in the set of resources that they provide access to because each of them is designed to fit a given type of job.&lt;br /&gt;
&lt;br /&gt;
Mufasa sets limits to the number of jobs by the same user. This page includes a [[#Limits on jobs by the same user|table summarising such limits]].&lt;br /&gt;
&lt;br /&gt;
= SLURM Quality of Service (QOS) =&lt;br /&gt;
&lt;br /&gt;
Through &amp;#039;&amp;#039;&amp;#039;Quality of Services&amp;#039;&amp;#039;&amp;#039; (&amp;#039;&amp;#039;&amp;#039;QOSes&amp;#039;&amp;#039;&amp;#039;), SLURM lets system configurators assign a name to a set of related constraints.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, QOSes are used to define different levels of access to the server&amp;#039;s resources. When [[User Jobs|executing a job with SLURM]], a user must always &amp;#039;&amp;#039;&amp;#039;specify the QOS&amp;#039;&amp;#039;&amp;#039; that their job will use: this choice, in turn, determines what resources the job is able to access and influences the [[#Job priority|priority]] of the job.&lt;br /&gt;
&lt;br /&gt;
Mufasa&amp;#039;s QOSes and their features can be inspected with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list qos format=name%-11,priority,MaxSubmit,maxwall,maxtres%-80,mintres%-18&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following (see below for the same data in table form):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
Name          Priority MaxSubmit     MaxWall MaxTRES                                                                          MinTRES            &lt;br /&gt;
----------- ---------- --------- ----------- -------------------------------------------------------------------------------- ------------------ &lt;br /&gt;
normal               0                                                                                                                           &lt;br /&gt;
nogpu                4         1  3-00:00:00 cpu=16,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=128G                    &lt;br /&gt;
gpuheavy-20          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=2,mem=128G             gres/gpu:4g.20gb=1 &lt;br /&gt;
gpuheavy-40          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=1,gres/gpu:4g.20gb=0,mem=128G             gres/gpu:40gb=1    &lt;br /&gt;
gpulight             8         1    12:00:00 cpu=2,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpu                  2         1  1-00:00:00 cpu=8,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpuwide              2         2  1-00:00:00 cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=1,mem=64G              gres/gpu:4g.20gb=1 &lt;br /&gt;
build               32         1    02:00:00 cpu=2,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=16G&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The columns of this output are the following:&lt;br /&gt;
&lt;br /&gt;
:; Name&lt;br /&gt;
:: name of the QOS&lt;br /&gt;
&lt;br /&gt;
:; Priority&lt;br /&gt;
:: priority tier associated to the QOS (higher value = higher priority): see [[#Job priority|Job priority]] for details&lt;br /&gt;
&lt;br /&gt;
:; MaxSubmit&lt;br /&gt;
:: maximum number of jobs from a single user that can be submitted to SLURM with this QOS; submitted jobs include both running and queued jobs&lt;br /&gt;
:: See [[#Limits on jobs by the same user|Limits on jobs by the same user]] for an overview of the limits on jobs set by Mufasa.&lt;br /&gt;
&lt;br /&gt;
:; MaxWall&lt;br /&gt;
:: maximum wall clock duration of the jobs using the QOS (after which they are killed by SLURM), in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
:: For some QOSes these are not set: it means that they are determined by the [[#SLURM partitions|partition]]. Partitions also define the [[#Default values|default duration]] of jobs.&lt;br /&gt;
&lt;br /&gt;
:; MaxTRES&lt;br /&gt;
:: amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) available to a job using the QOS, where&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;cpu=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of CPUs (i.e., processor cores) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of CPUs specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;gres/&amp;#039;&amp;#039;gpu:Type&amp;#039;&amp;#039;=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of GPUs of class &amp;lt;code&amp;gt;&amp;#039;&amp;#039;Type&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; (see [[User Jobs#gres syntax|&amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax]]) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; (for QOSes that allow access to GPUs) if not specified, the job cannot be launched&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;mem=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;G&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum amount of system RAM is &amp;#039;&amp;#039;K&amp;#039;&amp;#039; GBytes&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of RAM specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
&lt;br /&gt;
:; MinTRES&lt;br /&gt;
:: minimum amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) that a job using the QOS must request in order to actually get executed by SLURM.&lt;br /&gt;
:: (If your job does not actually need these resources, you&amp;#039;ve chosen the wrong QOS: you can use one with a higher priority.)&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; QOS is the default one, and exists only to ensure that users always specify a QOS when running a job. Since &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; has zero priority and no resources, a job run using this QOS would never be run.&lt;br /&gt;
&lt;br /&gt;
The information provided by the &amp;lt;code&amp;gt;sacctmgr list qos&amp;lt;/code&amp;gt; command above is summarised by the following table:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! QOS&lt;br /&gt;
! Priority&amp;lt;/br&amp;gt;tier&lt;br /&gt;
! Max&amp;lt;/br&amp;gt;Submit&lt;br /&gt;
! MaxWall&amp;lt;/br&amp;gt;[h]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;CPUs&lt;br /&gt;
! max RAM&amp;lt;/br&amp;gt;[GB]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;3g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;4g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;40GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | build&lt;br /&gt;
| 32&lt;br /&gt;
| 1&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 16&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpulight&lt;br /&gt;
| 8&lt;br /&gt;
| 1&lt;br /&gt;
| 12&lt;br /&gt;
| 2&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | nogpu&lt;br /&gt;
| 4&lt;br /&gt;
| 1&lt;br /&gt;
| 72&lt;br /&gt;
| 16&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpu&lt;br /&gt;
| 2&lt;br /&gt;
| 1&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuwide&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-20&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| 2 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-40&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
A key piece of information in the table above is the &amp;#039;&amp;#039;&amp;#039;priority tier&amp;#039;&amp;#039;&amp;#039; associated to each QOS. In Mufasa 2.0, priority tiers are used to encourage users to use the &amp;#039;&amp;#039;&amp;#039;least powerful QOS that is compatible with their needs&amp;#039;&amp;#039;&amp;#039;, where &amp;quot;powerful&amp;quot; means &amp;quot;rich with resources&amp;quot;. Less powerful QOSes increase the priority of the jobs that use them, so these jobs tend to be executed sooner. See [[#Job priority|Job priority]] for details about how priority affects the execution order of jobs in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
== The &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS ==&lt;br /&gt;
&lt;br /&gt;
This QOS is specifically designed to be used by Mufasa users to &amp;#039;&amp;#039;&amp;#039;quickly build [[System#Containers|container images]]&amp;#039;&amp;#039;&amp;#039;. Its associated priority tier is very high: this, combined with the fact that jobs using this QOS require few resources, means that such jobs usually get executed very soon.&lt;br /&gt;
&lt;br /&gt;
On the other hand, the limited resources and the lack of access to the GPUs make the &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS unsuitable for other tasks.&lt;br /&gt;
&lt;br /&gt;
See [[Singularity#Building Singularity images|Building Singularity images]] for directions about building Singularity container images.&lt;br /&gt;
&lt;br /&gt;
== Restricted QOSes ==&lt;br /&gt;
&lt;br /&gt;
In Mufasa, the most powerful QOSes are reserved to researchers (called &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users: these include academic personnel and Ph.D. students). M.Sc. students, called &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users, cannot use them while running jobs.&lt;br /&gt;
&lt;br /&gt;
See [[#research users and students users|below]] to understand how &amp;#039;&amp;#039;user categories&amp;#039;&amp;#039; work in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
= &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users =&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa belong to two &amp;#039;&amp;#039;&amp;#039;user categories&amp;#039;&amp;#039;&amp;#039;. Categories provide users with different access to  to Mufasa&amp;#039;s resources: the idea being to provide researchers with more access while still letting students benefit from the server&amp;#039;s capabilities.&lt;br /&gt;
&lt;br /&gt;
User categories in Mufasa are the following:&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. academic personnel and Ph.D. students&lt;br /&gt;
::: * can use all QOSes, including the the [[#Restricted QOSes|restricted ones]]&lt;br /&gt;
::: * their jobs have a higher &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a higher number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. M.Sc. students&lt;br /&gt;
::: * cannot use the [[#Restricted QOSes|restricted QOSes]]&lt;br /&gt;
::: * their jobs have a lower &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a lower number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
You can inspect the differences in priority and running jobs between &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association format=&amp;quot;account,priority,maxjobs&amp;quot; | grep -E &amp;#039;Account|research|students&amp;#039;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
   Account   Priority MaxJobs &lt;br /&gt;
  research          4       2 &lt;br /&gt;
  students          1       1&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
To know what limits apply to your own user, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association where user=$USER format=&amp;quot;user,priority,maxjobs,qos%-60&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
      User   Priority MaxJobs QOS                                                          &lt;br /&gt;
---------- ---------- ------- ------------------------------------------------------------ &lt;br /&gt;
    preali          4       2 build,gpu,gpuheavy-20,gpuheavy-40,gpulight,gpuwide,nogpu&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The list under &amp;quot;QOS&amp;quot; shows what QOSes your user is allowed to use when [[User Jobs|running jobs]]. &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users can use all of them, while &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users can only access a subset of them.&lt;br /&gt;
&lt;br /&gt;
= Limits on jobs by the same user =&lt;br /&gt;
&lt;br /&gt;
Mufasa enforces limits on the number of jobs from a single user. Such limits aim at preventing users from &amp;quot;hogging&amp;quot; system resources, and apply to:&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;submitted jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that the user asked SLURM to execute, each of which may currently be either running or queued&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;running jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that are currently in execution&lt;br /&gt;
&lt;br /&gt;
Limits on submitted jobs are set via [[#SLURM Quality of Service (QOS)|QOSes]], while limits on running jobs are set via [[#research users and students users|user category]].&lt;br /&gt;
&lt;br /&gt;
The following table summarises what limits exist:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! &lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;running&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;submitted&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | global limits&amp;lt;br/&amp;gt;(system-wide)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to limits set by individual QOSes (below)&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | limits for&amp;lt;br/&amp;gt;individual&amp;lt;/br&amp;gt;QOSes&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to global limits (above)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for the&amp;#039;&amp;#039; &amp;lt;code&amp;gt;gpuwide&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;QOS&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for each of the other QOSes&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
= Job priority =&lt;br /&gt;
&lt;br /&gt;
Once the execution of a job has been requested, the job is not run immediately: it is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039; by SLURM, together with all the other jobs awaiting execution. The job on top of the queue at any time is the first to be put into execution as soon as the resources it requires are available. &lt;br /&gt;
&lt;br /&gt;
The order of the items in the job queue depends on their &amp;#039;&amp;#039;&amp;#039;priority&amp;#039;&amp;#039;&amp;#039;. The job with the highest priority is put by SLURM on top of the queue, while all other jobs are put in the queue in order of descending priority.&lt;br /&gt;
&lt;br /&gt;
The goal of SLURM is to maximise resource availability: i.e., to ensure the shorter possible wait time before job execution. To achieve this goal, in Mufasa SLURM is configured to &amp;#039;&amp;#039;&amp;#039;encourage users not to ask for resources or execution time that their job doesn&amp;#039;t need&amp;#039;&amp;#039;&amp;#039;. This is done via the priority mechanism: the more resources and/or time a job requests, the lower its priority will be; and the later it will be executed.&lt;br /&gt;
&lt;br /&gt;
Priority management in Mufasa is designed to set up a &amp;#039;&amp;#039;&amp;#039;virtuous cycle&amp;#039;&amp;#039;&amp;#039; where users, by carefully choosing what to ask for, obtain two results:&lt;br /&gt;
* they ensure that their job is executed as soon as possible;&lt;br /&gt;
* they leave as much as possible of Mufasa&amp;#039;s resources free for other users&amp;#039;s jobs.&lt;br /&gt;
&lt;br /&gt;
== Elements determining job priority ==&lt;br /&gt;
In Mufasa, the priority of a job is computed by SLURM according to the following elements:&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#research users and students users|User category]]&amp;#039;&amp;#039;&amp;#039; (i.e., &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;)&lt;br /&gt;
::: Used to provide higher priority to jobs run by &amp;#039;&amp;#039;&amp;#039;research personnel&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#SLURM Quality of Service (QOS)|QOS]]&amp;#039;&amp;#039;&amp;#039; used by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;less resources&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Number of CPUs&amp;#039;&amp;#039;&amp;#039; requested by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;a lower number of CPUs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job Age&amp;#039;&amp;#039;&amp;#039;, i.e. the time that the job has been waiting in the queue&lt;br /&gt;
::: Used to provide higher priority to jobs which have been &amp;#039;&amp;#039;&amp;#039;queued for a longer time&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;FairShare&amp;#039;&amp;#039;&amp;#039;, i.e. a factor computed by SLURM to balance use of the system by different users&lt;br /&gt;
::: Used to provide higher priority to jobs by users who &amp;#039;&amp;#039;&amp;#039;used Mufasa&amp;#039;s resources less than others&amp;#039;&amp;#039;&amp;#039; (see [[#How FairShare works|below]])&lt;br /&gt;
&lt;br /&gt;
You can understand how each of these elements contributes to the priority of the jobs waiting for execution (both yours and of other users) with command &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;. A tailored version of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; that only displays the parameters listed above is the following:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sprio -o &amp;quot;%6i %9u %.5Y %.10B %.5Q %.9T %.5A %.5F&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Important!&lt;br /&gt;
:: The output of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; is only relevant for jobs &amp;#039;&amp;#039;&amp;#039;not yet in execution&amp;#039;&amp;#039;&amp;#039;. For an already running job, the value shown by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; is the priority that the job would get if computed now, which usually does not correspond to the actual priority.&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Here is an example of the output of the command above:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
JOBID  USER      PRIOR      ASSOC   QOS      TRES   AGE FAIRS&lt;br /&gt;
5774   itombini   3492       2500   750  cpu=-800   906   136&lt;br /&gt;
5780   fbottazzi  2266        625   750  cpu=-600   420  1071&lt;br /&gt;
5783   mzoccola   1766        625   750  cpu=-800   217   974&lt;br /&gt;
5790   amoglia    2717       2500   375  cpu=-800   156   487&lt;br /&gt;
5791   qliu       2232       2500   375  cpu=-800   138    19&lt;br /&gt;
5792   fpastori   2197       2500   375  cpu=-800    84    39&lt;br /&gt;
5793   bagostini  2034        625  1500 cpu=-1200    18  1091&lt;br /&gt;
5794   lpassoni   2517       2500   750  cpu=-800     9    58&lt;br /&gt;
5795   lpassoni   2517       2500   750  cpu=-800     9    58&lt;br /&gt;
5796   adurat     3101       2500   750  cpu=-800     8   643&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
In it, column ASSOC (i.e., association) corresponds to user category, while column TRES corresponds to the number of CPUs. Note that the contribution of CPUs to priority is &amp;#039;&amp;#039;negative&amp;#039;&amp;#039;, i.e. it is a penalty. Column PRIOR provides the overall priority of each job.&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
=== How FairShare works ===&lt;br /&gt;
In Mufasa 2.0, the FairShare of each user is computed according to these rules:&lt;br /&gt;
# It considers the quantity used by (current and past) user jobs of:&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;CPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of CPUs used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;RAM&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the amount of RAM used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;GPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of GPUs used&lt;br /&gt;
#** note: 40 GB GPUs have double the impact on FairShare than 20 GB GPUs&lt;br /&gt;
# The impact on FairShare of any use of resources is &amp;#039;&amp;#039;&amp;#039;proportional to its duration&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
#* example: using 32 GB of RAM for 48h has the same impact of using 64 GB of RAM for 24h&lt;br /&gt;
# FairShare has a &amp;quot;fading memory&amp;quot;: i.e., resource use has less and less impact on FairShare the farther in the past it is&lt;br /&gt;
#* consequence: over time, the &amp;quot;history&amp;quot; of a user gets forgotten by FairShare&lt;br /&gt;
&lt;br /&gt;
== How to maximise the priority of your jobs ==&lt;br /&gt;
&lt;br /&gt;
Every time you run a SLURM job, follow these guidelines:&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Choose the less powerful QOS compatible with the needs of your job&lt;br /&gt;
:: QOSes with access to less resources have higher base priority&lt;br /&gt;
&lt;br /&gt;
; Only request CPUs and RAM that your job really needs&lt;br /&gt;
:: Usually, code exploits multiple CPUs only if specifically designed to do so: if unsure, only ask for 1 CPU&lt;br /&gt;
:: The fewer resources your job asks for, the less it will wait before they become available&lt;br /&gt;
:: Asking for fewer CPUs and/or less RAM improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Do not request more time than your jobs needs to complete&lt;br /&gt;
:: Make a worst-case estimate and only ask for that duration&lt;br /&gt;
:: Asking for less of Mufasa&amp;#039;s time improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Debug and test code using less powerful QOSes before running it with more powerful QOSes&lt;br /&gt;
:: Your test jobs will get a higher priority and your FairShare will improve&lt;br /&gt;
&lt;br /&gt;
; Cancel jobs that you don&amp;#039;t need them anymore&lt;br /&gt;
:: Always use [[User_Jobs#Cancelling_a_job_with_scancel|scancel]] to delete completed (or crashed) jobs: your Fairshare will improve&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Protip: it&amp;#039;s a good idea to [[User Jobs#Looking for unused GPUs|check for unused GPUs]] before choosing what to request. Requesting a GPU that is currently idle will help your job get executed sooner.&lt;br /&gt;
&lt;br /&gt;
= SLURM partitions =&lt;br /&gt;
&lt;br /&gt;
Partitions are another mechanism provided by SLURM to create different levels of access to system resources. Since in Mufasa 2.0 access to resources is controlled via [[#SLURM Quality of Service (QOS)|QOSes]], partitions are not very relevant. (This is one of the main differences between Mufasa 1.0 and Mufasa 2.0.)&lt;br /&gt;
&lt;br /&gt;
Note, however, that the default values for some features of SLURM jobs (e.g., duration) are [[#Default values|set by the partition]].&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, there is a single SLURM partition, called &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt;, and all jobs run on it. The state of &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt; can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sinfo -o &amp;quot;%10P %5a %9T %11L %10l&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PARTITION  AVAIL STATE     DEFAULTTIME TIMELIMIT &lt;br /&gt;
jobs*      up    idle      1:00:00     3-00:00:00&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where columns correspond to the following information:&lt;br /&gt;
&lt;br /&gt;
:; PARTITION&lt;br /&gt;
:: name of the partition; the asterisks indicates that it&amp;#039;s the default one&lt;br /&gt;
&lt;br /&gt;
:; AVAIL&lt;br /&gt;
:: state/availability of the partition: see [[#Partition availability|below]]&lt;br /&gt;
&lt;br /&gt;
:; STATE&lt;br /&gt;
:: state (using [https://slurm.schedmd.com/sinfo.html#SECTION_NODE-STATE-CODES these codes])&lt;br /&gt;
:: typical values are &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;mixed&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that some of the resources are busy executing jobs while other are idle, and &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;allocated&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that all of the resources are in use&lt;br /&gt;
&lt;br /&gt;
:; DEFAULTTIME&lt;br /&gt;
:: default runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:; TIMELIMIT&lt;br /&gt;
:: maximum runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
The asterisk at the end of the partition name indicates the default partition, i.e. the one on which jobs which do not ask for a specific partition are run.&lt;br /&gt;
&lt;br /&gt;
Command &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; does not tell you about the &amp;#039;&amp;#039;jobs&amp;#039;&amp;#039; submitted to a partition. This information is obtained, instead, with [[User Jobs#Inspecting jobs with squeue|command &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;]].&lt;br /&gt;
&lt;br /&gt;
== Partition availability ==&lt;br /&gt;
&lt;br /&gt;
The most important information that &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; provides is the &amp;#039;&amp;#039;&amp;#039;availability&amp;#039;&amp;#039;&amp;#039; (also called &amp;#039;&amp;#039;&amp;#039;state&amp;#039;&amp;#039;&amp;#039;) of partitions. This is shown in column &amp;quot;AVAIL&amp;quot;. Possible partition states are:&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is available&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed as soon as resources allow&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is in the process of becoming unavailable (i.e., of entering the &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; state: see below)&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is unavailable&lt;br /&gt;
:: There are no running jobs&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
When a partition goes from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; no harm is done to running jobs. In a normally functioning SLURM system, the passage from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; happens only when no jobs are running on the partition. If (e.g., due to a malfunction) the passage happens with jobs still running, they get killed.&lt;br /&gt;
&lt;br /&gt;
A partition in state &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; requires intervention by a [[Roles|Job Administrator]] to be restored to &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
== Default values ==&lt;br /&gt;
&lt;br /&gt;
The features of SLURM partitions, including the &amp;#039;&amp;#039;&amp;#039;default values&amp;#039;&amp;#039;&amp;#039; which are applied to jobs that do not make explicit requests, can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scontrol show partition&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PartitionName=jobs&lt;br /&gt;
   AllowGroups=ALL AllowAccounts=ALL AllowQos=nogpu,gpulight,gpu,gpuwide,gpuheavy-20,gpuheavy-40&lt;br /&gt;
   AllocNodes=ALL Default=YES QoS=N/A&lt;br /&gt;
=&amp;gt; DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO&lt;br /&gt;
   MaxNodes=UNLIMITED MaxTime=3-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED&lt;br /&gt;
   Nodes=gn01&lt;br /&gt;
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO&lt;br /&gt;
   OverTimeLimit=NONE PreemptMode=OFF&lt;br /&gt;
   State=UP TotalCPUs=48 TotalNodes=1 SelectTypeParameters=NONE&lt;br /&gt;
   JobDefaults=(null)&lt;br /&gt;
=&amp;gt; DefMemPerNode=4096 MaxMemPerNode=UNLIMITED&lt;br /&gt;
   TRES=cpu=48,mem=1011435M,node=1,billing=49,gres/gpu=13,gres/gpu:3g.20gb=5,gres/gpu:40gb=3,gres/gpu:4g.20gb=5&lt;br /&gt;
   TRESBillingWeights=cpu=1.0,gres/gpu:3g.20gb=6.0,gres/gpu:4g.20gb=6.0,gres/gpu:40gb=6.0,mem=0.05g&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In the example, we have highlighted with  &amp;#039;&amp;#039;&amp;#039;=&amp;gt;&amp;#039;&amp;#039;&amp;#039;  the most relevant default values for Mufasa users, i.e.:&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefaultTime&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default execution time assigned to a job run on the partition (e.g., 1 hour)&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefMemPerNode&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default amount of RAM assigned to a job run on the partition (e.g., 4GB)&lt;br /&gt;
&lt;br /&gt;
= System resources subjected to limitations =&lt;br /&gt;
&lt;br /&gt;
In systems based on SLURM like Mufasa, &amp;#039;&amp;#039;&amp;#039;TRES (Trackable RESources)&amp;#039;&amp;#039;&amp;#039; are (from [https://slurm.schedmd.com/tres.html SLURM&amp;#039;s documentation] &amp;quot;&amp;#039;&amp;#039;resources that can be tracked for usage or used to enforce limits against.&amp;#039;&amp;#039;&amp;quot;&lt;br /&gt;
&lt;br /&gt;
TRES include CPUs, RAM and &amp;#039;&amp;#039;&amp;#039;GRES&amp;#039;&amp;#039;&amp;#039;. The last term stands for &amp;#039;&amp;#039;Generic RESources&amp;#039;&amp;#039; that a job may need for its execution. In Mufasa, the only &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; resources are the GPUs.&lt;br /&gt;
&lt;br /&gt;
== &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax ==&lt;br /&gt;
&lt;br /&gt;
To ask SLURM to assign GRES resources (i.e., GPUs) to a job, a special syntax must be used. Precisely, the name of each GPU resource takes the form&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:Name:Type&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
Considering the [[System#CPUs and GPUs|GPU complement of Mufasa]], &amp;lt;code&amp;gt;Type&amp;lt;/code&amp;gt; takes the following values:&lt;br /&gt;
&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:40gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 40 Gbytes of RAM&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:4g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 4 compute units&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 3 compute units&lt;br /&gt;
&lt;br /&gt;
So, for instance,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
identifies a resource corresponding to a GPU with 20 GB of RAM and 3 compute units.&lt;br /&gt;
&lt;br /&gt;
When asking for a GRES resource (e.g., in an &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command or an &amp;lt;code&amp;gt;SBATCH&amp;lt;/code&amp;gt; directive of an [[User Jobs#Using execution scripts to run jobs|execution script]]), the syntax required by SLURM is&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:&amp;lt;Type&amp;gt;:&amp;lt;Quantity&amp;gt;&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
where &amp;lt;code&amp;gt;Quantity&amp;lt;/code&amp;gt; is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of type &amp;lt;code&amp;gt;4g.20gb&amp;lt;/code&amp;gt; the syntax is&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:4g.20gb:2&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SLURM&amp;#039;s &amp;#039;&amp;#039;generic resources&amp;#039;&amp;#039; are defined in &amp;lt;code&amp;gt;/etc/slurm/gres.conf&amp;lt;/code&amp;gt;. In order to make GPUs available to SLURM&amp;#039;s &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; management, Mufasa makes use of Nvidia&amp;#039;s [https://developer.nvidia.com/nvidia-management-library-nvml NVML library]. For additional information see [https://slurm.schedmd.com/gres.html SLURM&amp;#039;s documentation].&lt;/div&gt;</summary>
		<author><name>GiulioFontana</name></author>
	</entry>
	<entry>
		<id>https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2579</id>
		<title>SLURM</title>
		<link rel="alternate" type="text/html" href="https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2579"/>
		<updated>2026-11-03T14:43:42Z</updated>

		<summary type="html">&lt;p&gt;GiulioFontana: /* Elements determining job priority */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page presents the features of SLURM that are most relevant to Mufasa&amp;#039;s [[Roles|Job Users]]. Job Users can submit jobs for execution, cancel their own jobs, and see other users&amp;#039; jobs (but not intervene on them).&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa &amp;#039;&amp;#039;&amp;#039;must use SLURM&amp;#039;&amp;#039;&amp;#039; to run resource-heavy processes, i.e. computing jobs that require one or more of the following:&lt;br /&gt;
* GPUs&lt;br /&gt;
* multiple CPUs&lt;br /&gt;
* powerful CPUs&lt;br /&gt;
* a significant amount of RAM&lt;br /&gt;
&lt;br /&gt;
In fact, only processes run via SLURM have access to all the resources of Mufasa. Processes run outside SLURM are executed by the [[System#Login server|login server]] virtual machine, which has minimal resources and no GPUs. Using SLURM is therefore the only way to execute resource-heavy jobs on Mufasa (this is a key difference between Mufasa 1.0 and Mufasa 2.0).&lt;br /&gt;
&lt;br /&gt;
= SLURM in a nutshell =&lt;br /&gt;
&lt;br /&gt;
Computation jobs on Mufasa needs to be launched via [[System#The SLURM job scheduling system|SLURM]]. SLURM provides jobs with access to the [[#System resources subjected to limitations|physical resources]] of Mufasa, such as CPUs, GPUs and RAM. Thanks to SLURM, processing jobs share system resources, optimising their occupation and availability. &lt;br /&gt;
&lt;br /&gt;
When a user runs a job, the job does not get executed immediately and is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039;. SLURM executes jobs according to their order in the queue: the top job in the queue gets executed as soon as the necessary resources are available, while jobs lower in the queue wait longer. The position of a job in the queue is due to the &amp;#039;&amp;#039;&amp;#039;[[#Job priority|priority]]&amp;#039;&amp;#039;&amp;#039; assigned to it by SLURM, with higher-priority jobs closer to the top. As a general rule:&lt;br /&gt;
&lt;br /&gt;
;: &amp;#039;&amp;#039;&amp;#039;the greater the fraction of Mufasa&amp;#039;s overall resources that a job asks for, the lower the job&amp;#039;s priority will be&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
The priority mechanism is used to encourage users to use Mufasa&amp;#039;s resources (i.e.: GPUs, CPUs, RAM, execution time) in an effective and equitable manner. This page includes a [[#How_to_maximise_the_priority_of_your_jobs|chart explaining how to maximise the priority of your jobs]].&lt;br /&gt;
&lt;br /&gt;
The &amp;#039;&amp;#039;&amp;#039;time&amp;#039;&amp;#039;&amp;#039; available to a job for its execution is controlled by SLURM. When a user requests execution of a job, they must specify the duration of the time slot that the job needs. The job must complete its execution before the end of the requested time slot, because at the end of such slot it gets killed by SLURM.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0 access to system resources is managed via SLURM&amp;#039;s [[#SLURM Quality of Service (QOS)|Quality of Service (QOS)]] mechanism (Mufasa 1.0 used [[#SLURM_partitions|partitions]] instead). While [[User Jobs#Running jobs with SLURM|launching a processing job via SLURM]], the user must always specify the chosen QOS. QOSes differ in the set of resources that they provide access to because each of them is designed to fit a given type of job.&lt;br /&gt;
&lt;br /&gt;
Mufasa sets limits to the number of jobs by the same user. This page includes a [[#Limits on jobs by the same user|table summarising such limits]].&lt;br /&gt;
&lt;br /&gt;
= SLURM Quality of Service (QOS) =&lt;br /&gt;
&lt;br /&gt;
Through &amp;#039;&amp;#039;&amp;#039;Quality of Services&amp;#039;&amp;#039;&amp;#039; (&amp;#039;&amp;#039;&amp;#039;QOSes&amp;#039;&amp;#039;&amp;#039;), SLURM lets system configurators assign a name to a set of related constraints.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, QOSes are used to define different levels of access to the server&amp;#039;s resources. When [[User Jobs|executing a job with SLURM]], a user must always &amp;#039;&amp;#039;&amp;#039;specify the QOS&amp;#039;&amp;#039;&amp;#039; that their job will use: this choice, in turn, determines what resources the job is able to access and influences the [[#Job priority|priority]] of the job.&lt;br /&gt;
&lt;br /&gt;
Mufasa&amp;#039;s QOSes and their features can be inspected with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list qos format=name%-11,priority,MaxSubmit,maxwall,maxtres%-80,mintres%-18&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following (see below for the same data in table form):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
Name          Priority MaxSubmit     MaxWall MaxTRES                                                                          MinTRES            &lt;br /&gt;
----------- ---------- --------- ----------- -------------------------------------------------------------------------------- ------------------ &lt;br /&gt;
normal               0                                                                                                                           &lt;br /&gt;
nogpu                4         1  3-00:00:00 cpu=16,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=128G                    &lt;br /&gt;
gpuheavy-20          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=2,mem=128G             gres/gpu:4g.20gb=1 &lt;br /&gt;
gpuheavy-40          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=1,gres/gpu:4g.20gb=0,mem=128G             gres/gpu:40gb=1    &lt;br /&gt;
gpulight             8         1    12:00:00 cpu=2,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpu                  2         1  1-00:00:00 cpu=8,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpuwide              2         2  1-00:00:00 cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=1,mem=64G              gres/gpu:4g.20gb=1 &lt;br /&gt;
build               32         1    02:00:00 cpu=2,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=16G&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The columns of this output are the following:&lt;br /&gt;
&lt;br /&gt;
:; Name&lt;br /&gt;
:: name of the QOS&lt;br /&gt;
&lt;br /&gt;
:; Priority&lt;br /&gt;
:: priority tier associated to the QOS (higher value = higher priority): see [[#Job priority|Job priority]] for details&lt;br /&gt;
&lt;br /&gt;
:; MaxSubmit&lt;br /&gt;
:: maximum number of jobs from a single user that can be submitted to SLURM with this QOS; submitted jobs include both running and queued jobs&lt;br /&gt;
:: See [[#Limits on jobs by the same user|Limits on jobs by the same user]] for an overview of the limits on jobs set by Mufasa.&lt;br /&gt;
&lt;br /&gt;
:; MaxWall&lt;br /&gt;
:: maximum wall clock duration of the jobs using the QOS (after which they are killed by SLURM), in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
:: For some QOSes these are not set: it means that they are determined by the [[#SLURM partitions|partition]]. Partitions also define the [[#Default values|default duration]] of jobs.&lt;br /&gt;
&lt;br /&gt;
:; MaxTRES&lt;br /&gt;
:: amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) available to a job using the QOS, where&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;cpu=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of CPUs (i.e., processor cores) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of CPUs specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;gres/&amp;#039;&amp;#039;gpu:Type&amp;#039;&amp;#039;=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of GPUs of class &amp;lt;code&amp;gt;&amp;#039;&amp;#039;Type&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; (see [[User Jobs#gres syntax|&amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax]]) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; (for QOSes that allow access to GPUs) if not specified, the job cannot be launched&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;mem=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;G&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum amount of system RAM is &amp;#039;&amp;#039;K&amp;#039;&amp;#039; GBytes&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of RAM specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
&lt;br /&gt;
:; MinTRES&lt;br /&gt;
:: minimum amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) that a job using the QOS must request in order to actually get executed by SLURM.&lt;br /&gt;
:: (If your job does not actually need these resources, you&amp;#039;ve chosen the wrong QOS: you can use one with a higher priority.)&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; QOS is the default one, and exists only to ensure that users always specify a QOS when running a job. Since &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; has zero priority and no resources, a job run using this QOS would never be run.&lt;br /&gt;
&lt;br /&gt;
The information provided by the &amp;lt;code&amp;gt;sacctmgr list qos&amp;lt;/code&amp;gt; command above is summarised by the following table:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! QOS&lt;br /&gt;
! Priority&amp;lt;/br&amp;gt;tier&lt;br /&gt;
! Max&amp;lt;/br&amp;gt;Submit&lt;br /&gt;
! MaxWall&amp;lt;/br&amp;gt;[h]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;CPUs&lt;br /&gt;
! max RAM&amp;lt;/br&amp;gt;[GB]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;3g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;4g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;40GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | build&lt;br /&gt;
| 32&lt;br /&gt;
| 1&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 16&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpulight&lt;br /&gt;
| 8&lt;br /&gt;
| 1&lt;br /&gt;
| 12&lt;br /&gt;
| 2&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | nogpu&lt;br /&gt;
| 4&lt;br /&gt;
| 1&lt;br /&gt;
| 72&lt;br /&gt;
| 16&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpu&lt;br /&gt;
| 2&lt;br /&gt;
| 1&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuwide&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-20&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| 2 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-40&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
A key piece of information in the table above is the &amp;#039;&amp;#039;&amp;#039;priority tier&amp;#039;&amp;#039;&amp;#039; associated to each QOS. In Mufasa 2.0, priority tiers are used to encourage users to use the &amp;#039;&amp;#039;&amp;#039;least powerful QOS that is compatible with their needs&amp;#039;&amp;#039;&amp;#039;, where &amp;quot;powerful&amp;quot; means &amp;quot;rich with resources&amp;quot;. Less powerful QOSes increase the priority of the jobs that use them, so these jobs tend to be executed sooner. See [[#Job priority|Job priority]] for details about how priority affects the execution order of jobs in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
== The &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS ==&lt;br /&gt;
&lt;br /&gt;
This QOS is specifically designed to be used by Mufasa users to &amp;#039;&amp;#039;&amp;#039;quickly build [[System#Containers|container images]]&amp;#039;&amp;#039;&amp;#039;. Its associated priority tier is very high: this, combined with the fact that jobs using this QOS require few resources, means that such jobs usually get executed very soon.&lt;br /&gt;
&lt;br /&gt;
On the other hand, the limited resources and the lack of access to the GPUs make the &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS unsuitable for other tasks.&lt;br /&gt;
&lt;br /&gt;
See [[Singularity#Building Singularity images|Building Singularity images]] for directions about building Singularity container images.&lt;br /&gt;
&lt;br /&gt;
== Restricted QOSes ==&lt;br /&gt;
&lt;br /&gt;
In Mufasa, the most powerful QOSes are reserved to researchers (called &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users: these include academic personnel and Ph.D. students). M.Sc. students, called &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users, cannot use them while running jobs.&lt;br /&gt;
&lt;br /&gt;
See [[#research users and students users|below]] to understand how &amp;#039;&amp;#039;user categories&amp;#039;&amp;#039; work in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
= &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users =&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa belong to two &amp;#039;&amp;#039;&amp;#039;user categories&amp;#039;&amp;#039;&amp;#039;. Categories provide users with different access to  to Mufasa&amp;#039;s resources: the idea being to provide researchers with more access while still letting students benefit from the server&amp;#039;s capabilities.&lt;br /&gt;
&lt;br /&gt;
User categories in Mufasa are the following:&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. academic personnel and Ph.D. students&lt;br /&gt;
::: * can use all QOSes, including the the [[#Restricted QOSes|restricted ones]]&lt;br /&gt;
::: * their jobs have a higher &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a higher number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. M.Sc. students&lt;br /&gt;
::: * cannot use the [[#Restricted QOSes|restricted QOSes]]&lt;br /&gt;
::: * their jobs have a lower &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a lower number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
You can inspect the differences in priority and running jobs between &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association format=&amp;quot;account,priority,maxjobs&amp;quot; | grep -E &amp;#039;Account|research|students&amp;#039;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
   Account   Priority MaxJobs &lt;br /&gt;
  research          4       2 &lt;br /&gt;
  students          1       1&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
To know what limits apply to your own user, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association where user=$USER format=&amp;quot;user,priority,maxjobs,qos%-60&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
      User   Priority MaxJobs QOS                                                          &lt;br /&gt;
---------- ---------- ------- ------------------------------------------------------------ &lt;br /&gt;
    preali          4       2 build,gpu,gpuheavy-20,gpuheavy-40,gpulight,gpuwide,nogpu&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The list under &amp;quot;QOS&amp;quot; shows what QOSes your user is allowed to use when [[User Jobs|running jobs]]. &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users can use all of them, while &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users can only access a subset of them.&lt;br /&gt;
&lt;br /&gt;
= Limits on jobs by the same user =&lt;br /&gt;
&lt;br /&gt;
Mufasa enforces limits on the number of jobs from a single user. Such limits aim at preventing users from &amp;quot;hogging&amp;quot; system resources, and apply to:&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;submitted jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that the user asked SLURM to execute, each of which may currently be either running or queued&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;running jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that are currently in execution&lt;br /&gt;
&lt;br /&gt;
Limits on submitted jobs are set via [[#SLURM Quality of Service (QOS)|QOSes]], while limits on running jobs are set via [[#research users and students users|user category]].&lt;br /&gt;
&lt;br /&gt;
The following table summarises what limits exist:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! &lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;running&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;submitted&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | global limits&amp;lt;br/&amp;gt;(system-wide)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to limits set by individual QOSes (below)&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | limits for&amp;lt;br/&amp;gt;individual&amp;lt;/br&amp;gt;QOSes&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to global limits (above)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for the&amp;#039;&amp;#039; &amp;lt;code&amp;gt;gpuwide&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;QOS&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for each of the other QOSes&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
= Job priority =&lt;br /&gt;
&lt;br /&gt;
Once the execution of a job has been requested, the job is not run immediately: it is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039; by SLURM, together with all the other jobs awaiting execution. The job on top of the queue at any time is the first to be put into execution as soon as the resources it requires are available. &lt;br /&gt;
&lt;br /&gt;
The order of the items in the job queue depends on their &amp;#039;&amp;#039;&amp;#039;priority&amp;#039;&amp;#039;&amp;#039;. The job with the highest priority is put by SLURM on top of the queue, while all other jobs are put in the queue in order of descending priority.&lt;br /&gt;
&lt;br /&gt;
The goal of SLURM is to maximise resource availability: i.e., to ensure the shorter possible wait time before job execution. To achieve this goal, in Mufasa SLURM is configured to &amp;#039;&amp;#039;&amp;#039;encourage users not to ask for resources or execution time that their job doesn&amp;#039;t need&amp;#039;&amp;#039;&amp;#039;. This is done via the priority mechanism: the more resources and/or time a job requests, the lower its priority will be; and the later it will be executed.&lt;br /&gt;
&lt;br /&gt;
Priority management in Mufasa is designed to set up a &amp;#039;&amp;#039;&amp;#039;virtuous cycle&amp;#039;&amp;#039;&amp;#039; where users, by carefully choosing what to ask for, obtain two results:&lt;br /&gt;
* they ensure that their job is executed as soon as possible;&lt;br /&gt;
* they leave as much as possible of Mufasa&amp;#039;s resources free for other users&amp;#039;s jobs.&lt;br /&gt;
&lt;br /&gt;
== Elements determining job priority ==&lt;br /&gt;
In Mufasa, the priority of a job is computed by SLURM according to the following elements:&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#research users and students users|User category]]&amp;#039;&amp;#039;&amp;#039; (i.e., &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;)&lt;br /&gt;
::: Used to provide higher priority to jobs run by &amp;#039;&amp;#039;&amp;#039;research personnel&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#SLURM Quality of Service (QOS)|QOS]]&amp;#039;&amp;#039;&amp;#039; used by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;less resources&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Number of CPUs&amp;#039;&amp;#039;&amp;#039; requested by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;a lower number of CPUs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job Age&amp;#039;&amp;#039;&amp;#039;, i.e. the time that the job has been waiting in the queue&lt;br /&gt;
::: Used to provide higher priority to jobs which have been &amp;#039;&amp;#039;&amp;#039;queued for a longer time&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;FairShare&amp;#039;&amp;#039;&amp;#039;, i.e. a factor computed by SLURM to balance use of the system by different users&lt;br /&gt;
::: Used to provide higher priority to jobs by users who &amp;#039;&amp;#039;&amp;#039;used Mufasa&amp;#039;s resources less than others&amp;#039;&amp;#039;&amp;#039; (see [[#How FairShare works|below]])&lt;br /&gt;
&lt;br /&gt;
You can understand how each of these elements contributes to the priority of the jobs waiting for execution (both yours and of other users) with command &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;. A tailored version of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; that only displays the parameters listed above is the following:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sprio -o &amp;quot;%6i %9u %.5Y %.10B %.5Q %.9T %.5A %.5F&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Important!&lt;br /&gt;
:: The output of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; is only relevant for jobs &amp;#039;&amp;#039;&amp;#039;not yet in execution&amp;#039;&amp;#039;&amp;#039;. For an already running job, the value shown by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; is the priority that the job would get if computed now, which usually does not correspond to the actual priority.&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Here is an example of the output of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
JOBID  USER      PRIOR      ASSOC   QOS      TRES   AGE FAIRS&lt;br /&gt;
5774   itombini   3492       2500   750  cpu=-800   906   136&lt;br /&gt;
5780   fbottazzi  2266        625   750  cpu=-600   420  1071&lt;br /&gt;
5783   mzoccola   1766        625   750  cpu=-800   217   974&lt;br /&gt;
5790   amoglia    2717       2500   375  cpu=-800   156   487&lt;br /&gt;
5791   qliu       2232       2500   375  cpu=-800   138    19&lt;br /&gt;
5792   fpastori   2197       2500   375  cpu=-800    84    39&lt;br /&gt;
5793   bagostini  2034        625  1500 cpu=-1200    18  1091&lt;br /&gt;
5794   lpassoni   2517       2500   750  cpu=-800     9    58&lt;br /&gt;
5795   lpassoni   2517       2500   750  cpu=-800     9    58&lt;br /&gt;
5796   adurat     3101       2500   750  cpu=-800     8   643&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
In it, column ASSOC (i.e., association) corresponds to user category, while column TRES corresponds to the number of CPUs. Note that the contribution of CPUs to priority is &amp;#039;&amp;#039;negative&amp;#039;&amp;#039;, i.e. it is a penalty. Column PRIOR provides the overall priority of each job.&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
=== How FairShare works ===&lt;br /&gt;
In Mufasa 2.0, the FairShare of each user is computed according to these rules:&lt;br /&gt;
# It considers the quantity used by (current and past) user jobs of:&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;CPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of CPUs used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;RAM&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the amount of RAM used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;GPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of GPUs used&lt;br /&gt;
#** note: 40 GB GPUs have double the impact on FairShare than 20 GB GPUs&lt;br /&gt;
# The impact on FairShare of any use of resources is &amp;#039;&amp;#039;&amp;#039;proportional to its duration&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
#* example: using 32 GB of RAM for 48h has the same impact of using 64 GB of RAM for 24h&lt;br /&gt;
# FairShare has a &amp;quot;fading memory&amp;quot;: i.e., resource use has less and less impact on FairShare the farther in the past it is&lt;br /&gt;
#* consequence: over time, the &amp;quot;history&amp;quot; of a user gets forgotten by FairShare&lt;br /&gt;
&lt;br /&gt;
== How to maximise the priority of your jobs ==&lt;br /&gt;
&lt;br /&gt;
Every time you run a SLURM job, follow these guidelines:&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Choose the less powerful QOS compatible with the needs of your job&lt;br /&gt;
:: QOSes with access to less resources have higher base priority&lt;br /&gt;
&lt;br /&gt;
; Only request CPUs and RAM that your job really needs&lt;br /&gt;
:: Usually, code exploits multiple CPUs only if specifically designed to do so: if unsure, only ask for 1 CPU&lt;br /&gt;
:: The fewer resources your job asks for, the less it will wait before they become available&lt;br /&gt;
:: Asking for fewer CPUs and/or less RAM improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Do not request more time than your jobs needs to complete&lt;br /&gt;
:: Make a worst-case estimate and only ask for that duration&lt;br /&gt;
:: Asking for less of Mufasa&amp;#039;s time improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Debug and test code using less powerful QOSes before running it with more powerful QOSes&lt;br /&gt;
:: Your test jobs will get a higher priority and your FairShare will improve&lt;br /&gt;
&lt;br /&gt;
; Cancel jobs that you don&amp;#039;t need them anymore&lt;br /&gt;
:: Always use [[User_Jobs#Cancelling_a_job_with_scancel|scancel]] to delete completed (or crashed) jobs: your Fairshare will improve&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Protip: it&amp;#039;s a good idea to [[User Jobs#Looking for unused GPUs|check for unused GPUs]] before choosing what to request. Requesting a GPU that is currently idle will help your job get executed sooner.&lt;br /&gt;
&lt;br /&gt;
= SLURM partitions =&lt;br /&gt;
&lt;br /&gt;
Partitions are another mechanism provided by SLURM to create different levels of access to system resources. Since in Mufasa 2.0 access to resources is controlled via [[#SLURM Quality of Service (QOS)|QOSes]], partitions are not very relevant. (This is one of the main differences between Mufasa 1.0 and Mufasa 2.0.)&lt;br /&gt;
&lt;br /&gt;
Note, however, that the default values for some features of SLURM jobs (e.g., duration) are [[#Default values|set by the partition]].&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, there is a single SLURM partition, called &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt;, and all jobs run on it. The state of &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt; can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sinfo -o &amp;quot;%10P %5a %9T %11L %10l&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PARTITION  AVAIL STATE     DEFAULTTIME TIMELIMIT &lt;br /&gt;
jobs*      up    idle      1:00:00     3-00:00:00&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where columns correspond to the following information:&lt;br /&gt;
&lt;br /&gt;
:; PARTITION&lt;br /&gt;
:: name of the partition; the asterisks indicates that it&amp;#039;s the default one&lt;br /&gt;
&lt;br /&gt;
:; AVAIL&lt;br /&gt;
:: state/availability of the partition: see [[#Partition availability|below]]&lt;br /&gt;
&lt;br /&gt;
:; STATE&lt;br /&gt;
:: state (using [https://slurm.schedmd.com/sinfo.html#SECTION_NODE-STATE-CODES these codes])&lt;br /&gt;
:: typical values are &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;mixed&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that some of the resources are busy executing jobs while other are idle, and &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;allocated&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that all of the resources are in use&lt;br /&gt;
&lt;br /&gt;
:; DEFAULTTIME&lt;br /&gt;
:: default runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:; TIMELIMIT&lt;br /&gt;
:: maximum runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
The asterisk at the end of the partition name indicates the default partition, i.e. the one on which jobs which do not ask for a specific partition are run.&lt;br /&gt;
&lt;br /&gt;
Command &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; does not tell you about the &amp;#039;&amp;#039;jobs&amp;#039;&amp;#039; submitted to a partition. This information is obtained, instead, with [[User Jobs#Inspecting jobs with squeue|command &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;]].&lt;br /&gt;
&lt;br /&gt;
== Partition availability ==&lt;br /&gt;
&lt;br /&gt;
The most important information that &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; provides is the &amp;#039;&amp;#039;&amp;#039;availability&amp;#039;&amp;#039;&amp;#039; (also called &amp;#039;&amp;#039;&amp;#039;state&amp;#039;&amp;#039;&amp;#039;) of partitions. This is shown in column &amp;quot;AVAIL&amp;quot;. Possible partition states are:&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is available&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed as soon as resources allow&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is in the process of becoming unavailable (i.e., of entering the &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; state: see below)&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is unavailable&lt;br /&gt;
:: There are no running jobs&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
When a partition goes from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; no harm is done to running jobs. In a normally functioning SLURM system, the passage from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; happens only when no jobs are running on the partition. If (e.g., due to a malfunction) the passage happens with jobs still running, they get killed.&lt;br /&gt;
&lt;br /&gt;
A partition in state &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; requires intervention by a [[Roles|Job Administrator]] to be restored to &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
== Default values ==&lt;br /&gt;
&lt;br /&gt;
The features of SLURM partitions, including the &amp;#039;&amp;#039;&amp;#039;default values&amp;#039;&amp;#039;&amp;#039; which are applied to jobs that do not make explicit requests, can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scontrol show partition&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PartitionName=jobs&lt;br /&gt;
   AllowGroups=ALL AllowAccounts=ALL AllowQos=nogpu,gpulight,gpu,gpuwide,gpuheavy-20,gpuheavy-40&lt;br /&gt;
   AllocNodes=ALL Default=YES QoS=N/A&lt;br /&gt;
=&amp;gt; DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO&lt;br /&gt;
   MaxNodes=UNLIMITED MaxTime=3-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED&lt;br /&gt;
   Nodes=gn01&lt;br /&gt;
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO&lt;br /&gt;
   OverTimeLimit=NONE PreemptMode=OFF&lt;br /&gt;
   State=UP TotalCPUs=48 TotalNodes=1 SelectTypeParameters=NONE&lt;br /&gt;
   JobDefaults=(null)&lt;br /&gt;
=&amp;gt; DefMemPerNode=4096 MaxMemPerNode=UNLIMITED&lt;br /&gt;
   TRES=cpu=48,mem=1011435M,node=1,billing=49,gres/gpu=13,gres/gpu:3g.20gb=5,gres/gpu:40gb=3,gres/gpu:4g.20gb=5&lt;br /&gt;
   TRESBillingWeights=cpu=1.0,gres/gpu:3g.20gb=6.0,gres/gpu:4g.20gb=6.0,gres/gpu:40gb=6.0,mem=0.05g&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In the example, we have highlighted with  &amp;#039;&amp;#039;&amp;#039;=&amp;gt;&amp;#039;&amp;#039;&amp;#039;  the most relevant default values for Mufasa users, i.e.:&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefaultTime&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default execution time assigned to a job run on the partition (e.g., 1 hour)&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefMemPerNode&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default amount of RAM assigned to a job run on the partition (e.g., 4GB)&lt;br /&gt;
&lt;br /&gt;
= System resources subjected to limitations =&lt;br /&gt;
&lt;br /&gt;
In systems based on SLURM like Mufasa, &amp;#039;&amp;#039;&amp;#039;TRES (Trackable RESources)&amp;#039;&amp;#039;&amp;#039; are (from [https://slurm.schedmd.com/tres.html SLURM&amp;#039;s documentation] &amp;quot;&amp;#039;&amp;#039;resources that can be tracked for usage or used to enforce limits against.&amp;#039;&amp;#039;&amp;quot;&lt;br /&gt;
&lt;br /&gt;
TRES include CPUs, RAM and &amp;#039;&amp;#039;&amp;#039;GRES&amp;#039;&amp;#039;&amp;#039;. The last term stands for &amp;#039;&amp;#039;Generic RESources&amp;#039;&amp;#039; that a job may need for its execution. In Mufasa, the only &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; resources are the GPUs.&lt;br /&gt;
&lt;br /&gt;
== &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax ==&lt;br /&gt;
&lt;br /&gt;
To ask SLURM to assign GRES resources (i.e., GPUs) to a job, a special syntax must be used. Precisely, the name of each GPU resource takes the form&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:Name:Type&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
Considering the [[System#CPUs and GPUs|GPU complement of Mufasa]], &amp;lt;code&amp;gt;Type&amp;lt;/code&amp;gt; takes the following values:&lt;br /&gt;
&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:40gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 40 Gbytes of RAM&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:4g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 4 compute units&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 3 compute units&lt;br /&gt;
&lt;br /&gt;
So, for instance,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
identifies a resource corresponding to a GPU with 20 GB of RAM and 3 compute units.&lt;br /&gt;
&lt;br /&gt;
When asking for a GRES resource (e.g., in an &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command or an &amp;lt;code&amp;gt;SBATCH&amp;lt;/code&amp;gt; directive of an [[User Jobs#Using execution scripts to run jobs|execution script]]), the syntax required by SLURM is&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:&amp;lt;Type&amp;gt;:&amp;lt;Quantity&amp;gt;&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
where &amp;lt;code&amp;gt;Quantity&amp;lt;/code&amp;gt; is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of type &amp;lt;code&amp;gt;4g.20gb&amp;lt;/code&amp;gt; the syntax is&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:4g.20gb:2&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SLURM&amp;#039;s &amp;#039;&amp;#039;generic resources&amp;#039;&amp;#039; are defined in &amp;lt;code&amp;gt;/etc/slurm/gres.conf&amp;lt;/code&amp;gt;. In order to make GPUs available to SLURM&amp;#039;s &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; management, Mufasa makes use of Nvidia&amp;#039;s [https://developer.nvidia.com/nvidia-management-library-nvml NVML library]. For additional information see [https://slurm.schedmd.com/gres.html SLURM&amp;#039;s documentation].&lt;/div&gt;</summary>
		<author><name>GiulioFontana</name></author>
	</entry>
	<entry>
		<id>https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2578</id>
		<title>SLURM</title>
		<link rel="alternate" type="text/html" href="https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2578"/>
		<updated>2026-11-03T14:43:04Z</updated>

		<summary type="html">&lt;p&gt;GiulioFontana: /* Elements determining job priority */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page presents the features of SLURM that are most relevant to Mufasa&amp;#039;s [[Roles|Job Users]]. Job Users can submit jobs for execution, cancel their own jobs, and see other users&amp;#039; jobs (but not intervene on them).&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa &amp;#039;&amp;#039;&amp;#039;must use SLURM&amp;#039;&amp;#039;&amp;#039; to run resource-heavy processes, i.e. computing jobs that require one or more of the following:&lt;br /&gt;
* GPUs&lt;br /&gt;
* multiple CPUs&lt;br /&gt;
* powerful CPUs&lt;br /&gt;
* a significant amount of RAM&lt;br /&gt;
&lt;br /&gt;
In fact, only processes run via SLURM have access to all the resources of Mufasa. Processes run outside SLURM are executed by the [[System#Login server|login server]] virtual machine, which has minimal resources and no GPUs. Using SLURM is therefore the only way to execute resource-heavy jobs on Mufasa (this is a key difference between Mufasa 1.0 and Mufasa 2.0).&lt;br /&gt;
&lt;br /&gt;
= SLURM in a nutshell =&lt;br /&gt;
&lt;br /&gt;
Computation jobs on Mufasa needs to be launched via [[System#The SLURM job scheduling system|SLURM]]. SLURM provides jobs with access to the [[#System resources subjected to limitations|physical resources]] of Mufasa, such as CPUs, GPUs and RAM. Thanks to SLURM, processing jobs share system resources, optimising their occupation and availability. &lt;br /&gt;
&lt;br /&gt;
When a user runs a job, the job does not get executed immediately and is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039;. SLURM executes jobs according to their order in the queue: the top job in the queue gets executed as soon as the necessary resources are available, while jobs lower in the queue wait longer. The position of a job in the queue is due to the &amp;#039;&amp;#039;&amp;#039;[[#Job priority|priority]]&amp;#039;&amp;#039;&amp;#039; assigned to it by SLURM, with higher-priority jobs closer to the top. As a general rule:&lt;br /&gt;
&lt;br /&gt;
;: &amp;#039;&amp;#039;&amp;#039;the greater the fraction of Mufasa&amp;#039;s overall resources that a job asks for, the lower the job&amp;#039;s priority will be&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
The priority mechanism is used to encourage users to use Mufasa&amp;#039;s resources (i.e.: GPUs, CPUs, RAM, execution time) in an effective and equitable manner. This page includes a [[#How_to_maximise_the_priority_of_your_jobs|chart explaining how to maximise the priority of your jobs]].&lt;br /&gt;
&lt;br /&gt;
The &amp;#039;&amp;#039;&amp;#039;time&amp;#039;&amp;#039;&amp;#039; available to a job for its execution is controlled by SLURM. When a user requests execution of a job, they must specify the duration of the time slot that the job needs. The job must complete its execution before the end of the requested time slot, because at the end of such slot it gets killed by SLURM.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0 access to system resources is managed via SLURM&amp;#039;s [[#SLURM Quality of Service (QOS)|Quality of Service (QOS)]] mechanism (Mufasa 1.0 used [[#SLURM_partitions|partitions]] instead). While [[User Jobs#Running jobs with SLURM|launching a processing job via SLURM]], the user must always specify the chosen QOS. QOSes differ in the set of resources that they provide access to because each of them is designed to fit a given type of job.&lt;br /&gt;
&lt;br /&gt;
Mufasa sets limits to the number of jobs by the same user. This page includes a [[#Limits on jobs by the same user|table summarising such limits]].&lt;br /&gt;
&lt;br /&gt;
= SLURM Quality of Service (QOS) =&lt;br /&gt;
&lt;br /&gt;
Through &amp;#039;&amp;#039;&amp;#039;Quality of Services&amp;#039;&amp;#039;&amp;#039; (&amp;#039;&amp;#039;&amp;#039;QOSes&amp;#039;&amp;#039;&amp;#039;), SLURM lets system configurators assign a name to a set of related constraints.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, QOSes are used to define different levels of access to the server&amp;#039;s resources. When [[User Jobs|executing a job with SLURM]], a user must always &amp;#039;&amp;#039;&amp;#039;specify the QOS&amp;#039;&amp;#039;&amp;#039; that their job will use: this choice, in turn, determines what resources the job is able to access and influences the [[#Job priority|priority]] of the job.&lt;br /&gt;
&lt;br /&gt;
Mufasa&amp;#039;s QOSes and their features can be inspected with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list qos format=name%-11,priority,MaxSubmit,maxwall,maxtres%-80,mintres%-18&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following (see below for the same data in table form):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
Name          Priority MaxSubmit     MaxWall MaxTRES                                                                          MinTRES            &lt;br /&gt;
----------- ---------- --------- ----------- -------------------------------------------------------------------------------- ------------------ &lt;br /&gt;
normal               0                                                                                                                           &lt;br /&gt;
nogpu                4         1  3-00:00:00 cpu=16,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=128G                    &lt;br /&gt;
gpuheavy-20          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=2,mem=128G             gres/gpu:4g.20gb=1 &lt;br /&gt;
gpuheavy-40          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=1,gres/gpu:4g.20gb=0,mem=128G             gres/gpu:40gb=1    &lt;br /&gt;
gpulight             8         1    12:00:00 cpu=2,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpu                  2         1  1-00:00:00 cpu=8,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpuwide              2         2  1-00:00:00 cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=1,mem=64G              gres/gpu:4g.20gb=1 &lt;br /&gt;
build               32         1    02:00:00 cpu=2,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=16G&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The columns of this output are the following:&lt;br /&gt;
&lt;br /&gt;
:; Name&lt;br /&gt;
:: name of the QOS&lt;br /&gt;
&lt;br /&gt;
:; Priority&lt;br /&gt;
:: priority tier associated to the QOS (higher value = higher priority): see [[#Job priority|Job priority]] for details&lt;br /&gt;
&lt;br /&gt;
:; MaxSubmit&lt;br /&gt;
:: maximum number of jobs from a single user that can be submitted to SLURM with this QOS; submitted jobs include both running and queued jobs&lt;br /&gt;
:: See [[#Limits on jobs by the same user|Limits on jobs by the same user]] for an overview of the limits on jobs set by Mufasa.&lt;br /&gt;
&lt;br /&gt;
:; MaxWall&lt;br /&gt;
:: maximum wall clock duration of the jobs using the QOS (after which they are killed by SLURM), in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
:: For some QOSes these are not set: it means that they are determined by the [[#SLURM partitions|partition]]. Partitions also define the [[#Default values|default duration]] of jobs.&lt;br /&gt;
&lt;br /&gt;
:; MaxTRES&lt;br /&gt;
:: amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) available to a job using the QOS, where&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;cpu=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of CPUs (i.e., processor cores) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of CPUs specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;gres/&amp;#039;&amp;#039;gpu:Type&amp;#039;&amp;#039;=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of GPUs of class &amp;lt;code&amp;gt;&amp;#039;&amp;#039;Type&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; (see [[User Jobs#gres syntax|&amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax]]) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; (for QOSes that allow access to GPUs) if not specified, the job cannot be launched&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;mem=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;G&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum amount of system RAM is &amp;#039;&amp;#039;K&amp;#039;&amp;#039; GBytes&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of RAM specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
&lt;br /&gt;
:; MinTRES&lt;br /&gt;
:: minimum amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) that a job using the QOS must request in order to actually get executed by SLURM.&lt;br /&gt;
:: (If your job does not actually need these resources, you&amp;#039;ve chosen the wrong QOS: you can use one with a higher priority.)&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; QOS is the default one, and exists only to ensure that users always specify a QOS when running a job. Since &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; has zero priority and no resources, a job run using this QOS would never be run.&lt;br /&gt;
&lt;br /&gt;
The information provided by the &amp;lt;code&amp;gt;sacctmgr list qos&amp;lt;/code&amp;gt; command above is summarised by the following table:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! QOS&lt;br /&gt;
! Priority&amp;lt;/br&amp;gt;tier&lt;br /&gt;
! Max&amp;lt;/br&amp;gt;Submit&lt;br /&gt;
! MaxWall&amp;lt;/br&amp;gt;[h]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;CPUs&lt;br /&gt;
! max RAM&amp;lt;/br&amp;gt;[GB]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;3g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;4g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;40GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | build&lt;br /&gt;
| 32&lt;br /&gt;
| 1&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 16&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpulight&lt;br /&gt;
| 8&lt;br /&gt;
| 1&lt;br /&gt;
| 12&lt;br /&gt;
| 2&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | nogpu&lt;br /&gt;
| 4&lt;br /&gt;
| 1&lt;br /&gt;
| 72&lt;br /&gt;
| 16&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpu&lt;br /&gt;
| 2&lt;br /&gt;
| 1&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuwide&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-20&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| 2 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-40&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
A key piece of information in the table above is the &amp;#039;&amp;#039;&amp;#039;priority tier&amp;#039;&amp;#039;&amp;#039; associated to each QOS. In Mufasa 2.0, priority tiers are used to encourage users to use the &amp;#039;&amp;#039;&amp;#039;least powerful QOS that is compatible with their needs&amp;#039;&amp;#039;&amp;#039;, where &amp;quot;powerful&amp;quot; means &amp;quot;rich with resources&amp;quot;. Less powerful QOSes increase the priority of the jobs that use them, so these jobs tend to be executed sooner. See [[#Job priority|Job priority]] for details about how priority affects the execution order of jobs in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
== The &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS ==&lt;br /&gt;
&lt;br /&gt;
This QOS is specifically designed to be used by Mufasa users to &amp;#039;&amp;#039;&amp;#039;quickly build [[System#Containers|container images]]&amp;#039;&amp;#039;&amp;#039;. Its associated priority tier is very high: this, combined with the fact that jobs using this QOS require few resources, means that such jobs usually get executed very soon.&lt;br /&gt;
&lt;br /&gt;
On the other hand, the limited resources and the lack of access to the GPUs make the &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS unsuitable for other tasks.&lt;br /&gt;
&lt;br /&gt;
See [[Singularity#Building Singularity images|Building Singularity images]] for directions about building Singularity container images.&lt;br /&gt;
&lt;br /&gt;
== Restricted QOSes ==&lt;br /&gt;
&lt;br /&gt;
In Mufasa, the most powerful QOSes are reserved to researchers (called &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users: these include academic personnel and Ph.D. students). M.Sc. students, called &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users, cannot use them while running jobs.&lt;br /&gt;
&lt;br /&gt;
See [[#research users and students users|below]] to understand how &amp;#039;&amp;#039;user categories&amp;#039;&amp;#039; work in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
= &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users =&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa belong to two &amp;#039;&amp;#039;&amp;#039;user categories&amp;#039;&amp;#039;&amp;#039;. Categories provide users with different access to  to Mufasa&amp;#039;s resources: the idea being to provide researchers with more access while still letting students benefit from the server&amp;#039;s capabilities.&lt;br /&gt;
&lt;br /&gt;
User categories in Mufasa are the following:&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. academic personnel and Ph.D. students&lt;br /&gt;
::: * can use all QOSes, including the the [[#Restricted QOSes|restricted ones]]&lt;br /&gt;
::: * their jobs have a higher &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a higher number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. M.Sc. students&lt;br /&gt;
::: * cannot use the [[#Restricted QOSes|restricted QOSes]]&lt;br /&gt;
::: * their jobs have a lower &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a lower number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
You can inspect the differences in priority and running jobs between &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association format=&amp;quot;account,priority,maxjobs&amp;quot; | grep -E &amp;#039;Account|research|students&amp;#039;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
   Account   Priority MaxJobs &lt;br /&gt;
  research          4       2 &lt;br /&gt;
  students          1       1&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
To know what limits apply to your own user, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association where user=$USER format=&amp;quot;user,priority,maxjobs,qos%-60&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
      User   Priority MaxJobs QOS                                                          &lt;br /&gt;
---------- ---------- ------- ------------------------------------------------------------ &lt;br /&gt;
    preali          4       2 build,gpu,gpuheavy-20,gpuheavy-40,gpulight,gpuwide,nogpu&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The list under &amp;quot;QOS&amp;quot; shows what QOSes your user is allowed to use when [[User Jobs|running jobs]]. &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users can use all of them, while &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users can only access a subset of them.&lt;br /&gt;
&lt;br /&gt;
= Limits on jobs by the same user =&lt;br /&gt;
&lt;br /&gt;
Mufasa enforces limits on the number of jobs from a single user. Such limits aim at preventing users from &amp;quot;hogging&amp;quot; system resources, and apply to:&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;submitted jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that the user asked SLURM to execute, each of which may currently be either running or queued&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;running jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that are currently in execution&lt;br /&gt;
&lt;br /&gt;
Limits on submitted jobs are set via [[#SLURM Quality of Service (QOS)|QOSes]], while limits on running jobs are set via [[#research users and students users|user category]].&lt;br /&gt;
&lt;br /&gt;
The following table summarises what limits exist:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! &lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;running&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;submitted&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | global limits&amp;lt;br/&amp;gt;(system-wide)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to limits set by individual QOSes (below)&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | limits for&amp;lt;br/&amp;gt;individual&amp;lt;/br&amp;gt;QOSes&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to global limits (above)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for the&amp;#039;&amp;#039; &amp;lt;code&amp;gt;gpuwide&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;QOS&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for each of the other QOSes&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
= Job priority =&lt;br /&gt;
&lt;br /&gt;
Once the execution of a job has been requested, the job is not run immediately: it is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039; by SLURM, together with all the other jobs awaiting execution. The job on top of the queue at any time is the first to be put into execution as soon as the resources it requires are available. &lt;br /&gt;
&lt;br /&gt;
The order of the items in the job queue depends on their &amp;#039;&amp;#039;&amp;#039;priority&amp;#039;&amp;#039;&amp;#039;. The job with the highest priority is put by SLURM on top of the queue, while all other jobs are put in the queue in order of descending priority.&lt;br /&gt;
&lt;br /&gt;
The goal of SLURM is to maximise resource availability: i.e., to ensure the shorter possible wait time before job execution. To achieve this goal, in Mufasa SLURM is configured to &amp;#039;&amp;#039;&amp;#039;encourage users not to ask for resources or execution time that their job doesn&amp;#039;t need&amp;#039;&amp;#039;&amp;#039;. This is done via the priority mechanism: the more resources and/or time a job requests, the lower its priority will be; and the later it will be executed.&lt;br /&gt;
&lt;br /&gt;
Priority management in Mufasa is designed to set up a &amp;#039;&amp;#039;&amp;#039;virtuous cycle&amp;#039;&amp;#039;&amp;#039; where users, by carefully choosing what to ask for, obtain two results:&lt;br /&gt;
* they ensure that their job is executed as soon as possible;&lt;br /&gt;
* they leave as much as possible of Mufasa&amp;#039;s resources free for other users&amp;#039;s jobs.&lt;br /&gt;
&lt;br /&gt;
== Elements determining job priority ==&lt;br /&gt;
In Mufasa, the priority of a job is computed by SLURM according to the following elements:&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#research users and students users|User category]]&amp;#039;&amp;#039;&amp;#039; (i.e., &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;)&lt;br /&gt;
::: Used to provide higher priority to jobs run by &amp;#039;&amp;#039;&amp;#039;research personnel&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#SLURM Quality of Service (QOS)|QOS]]&amp;#039;&amp;#039;&amp;#039; used by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;less resources&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Number of CPUs&amp;#039;&amp;#039;&amp;#039; requested by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;a lower number of CPUs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job Age&amp;#039;&amp;#039;&amp;#039;, i.e. the time that the job has been waiting in the queue&lt;br /&gt;
::: Used to provide higher priority to jobs which have been &amp;#039;&amp;#039;&amp;#039;queued for a longer time&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;FairShare&amp;#039;&amp;#039;&amp;#039;, i.e. a factor computed by SLURM to balance use of the system by different users&lt;br /&gt;
::: Used to provide higher priority to jobs by users who &amp;#039;&amp;#039;&amp;#039;used Mufasa&amp;#039;s resources less than others&amp;#039;&amp;#039;&amp;#039; (see [[#How FairShare works|below]])&lt;br /&gt;
&lt;br /&gt;
You can understand how each of these elements contributes to the priority of the jobs waiting for execution (both yours and of other users) with command &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;. A tailored version of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; that only displays the parameters listed above is the following:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sprio -o &amp;quot;%6i %9u %.5Y %.10B %.5Q %.9T %.5A %.5F&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Important!&lt;br /&gt;
:: The output of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; is only relevant for jobs &amp;#039;&amp;#039;&amp;#039;not yet in execution&amp;#039;&amp;#039;&amp;#039;. For an already running job, the value shown by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; is the priority that the job would get if computed now, which usually does not correspond to the actual priority.&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Here is an example of its output:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
JOBID  USER      PRIOR      ASSOC   QOS      TRES   AGE FAIRS&lt;br /&gt;
5774   itombini   3492       2500   750  cpu=-800   906   136&lt;br /&gt;
5780   fbottazzi  2266        625   750  cpu=-600   420  1071&lt;br /&gt;
5783   mzoccola   1766        625   750  cpu=-800   217   974&lt;br /&gt;
5790   amoglia    2717       2500   375  cpu=-800   156   487&lt;br /&gt;
5791   qliu       2232       2500   375  cpu=-800   138    19&lt;br /&gt;
5792   fpastori   2197       2500   375  cpu=-800    84    39&lt;br /&gt;
5793   bagostini  2034        625  1500 cpu=-1200    18  1091&lt;br /&gt;
5794   lpassoni   2517       2500   750  cpu=-800     9    58&lt;br /&gt;
5795   lpassoni   2517       2500   750  cpu=-800     9    58&lt;br /&gt;
5796   adurat     3101       2500   750  cpu=-800     8   643&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
In it, column ASSOC (i.e., association) corresponds to user category, while column TRES corresponds to the number of CPUs. Note that the contribution of CPUs to priority is &amp;#039;&amp;#039;negative&amp;#039;&amp;#039;, i.e. it is a penalty. Column PRIOR provides the overall priority of each job.&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
=== How FairShare works ===&lt;br /&gt;
In Mufasa 2.0, the FairShare of each user is computed according to these rules:&lt;br /&gt;
# It considers the quantity used by (current and past) user jobs of:&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;CPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of CPUs used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;RAM&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the amount of RAM used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;GPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of GPUs used&lt;br /&gt;
#** note: 40 GB GPUs have double the impact on FairShare than 20 GB GPUs&lt;br /&gt;
# The impact on FairShare of any use of resources is &amp;#039;&amp;#039;&amp;#039;proportional to its duration&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
#* example: using 32 GB of RAM for 48h has the same impact of using 64 GB of RAM for 24h&lt;br /&gt;
# FairShare has a &amp;quot;fading memory&amp;quot;: i.e., resource use has less and less impact on FairShare the farther in the past it is&lt;br /&gt;
#* consequence: over time, the &amp;quot;history&amp;quot; of a user gets forgotten by FairShare&lt;br /&gt;
&lt;br /&gt;
== How to maximise the priority of your jobs ==&lt;br /&gt;
&lt;br /&gt;
Every time you run a SLURM job, follow these guidelines:&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Choose the less powerful QOS compatible with the needs of your job&lt;br /&gt;
:: QOSes with access to less resources have higher base priority&lt;br /&gt;
&lt;br /&gt;
; Only request CPUs and RAM that your job really needs&lt;br /&gt;
:: Usually, code exploits multiple CPUs only if specifically designed to do so: if unsure, only ask for 1 CPU&lt;br /&gt;
:: The fewer resources your job asks for, the less it will wait before they become available&lt;br /&gt;
:: Asking for fewer CPUs and/or less RAM improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Do not request more time than your jobs needs to complete&lt;br /&gt;
:: Make a worst-case estimate and only ask for that duration&lt;br /&gt;
:: Asking for less of Mufasa&amp;#039;s time improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Debug and test code using less powerful QOSes before running it with more powerful QOSes&lt;br /&gt;
:: Your test jobs will get a higher priority and your FairShare will improve&lt;br /&gt;
&lt;br /&gt;
; Cancel jobs that you don&amp;#039;t need them anymore&lt;br /&gt;
:: Always use [[User_Jobs#Cancelling_a_job_with_scancel|scancel]] to delete completed (or crashed) jobs: your Fairshare will improve&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Protip: it&amp;#039;s a good idea to [[User Jobs#Looking for unused GPUs|check for unused GPUs]] before choosing what to request. Requesting a GPU that is currently idle will help your job get executed sooner.&lt;br /&gt;
&lt;br /&gt;
= SLURM partitions =&lt;br /&gt;
&lt;br /&gt;
Partitions are another mechanism provided by SLURM to create different levels of access to system resources. Since in Mufasa 2.0 access to resources is controlled via [[#SLURM Quality of Service (QOS)|QOSes]], partitions are not very relevant. (This is one of the main differences between Mufasa 1.0 and Mufasa 2.0.)&lt;br /&gt;
&lt;br /&gt;
Note, however, that the default values for some features of SLURM jobs (e.g., duration) are [[#Default values|set by the partition]].&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, there is a single SLURM partition, called &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt;, and all jobs run on it. The state of &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt; can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sinfo -o &amp;quot;%10P %5a %9T %11L %10l&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PARTITION  AVAIL STATE     DEFAULTTIME TIMELIMIT &lt;br /&gt;
jobs*      up    idle      1:00:00     3-00:00:00&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where columns correspond to the following information:&lt;br /&gt;
&lt;br /&gt;
:; PARTITION&lt;br /&gt;
:: name of the partition; the asterisks indicates that it&amp;#039;s the default one&lt;br /&gt;
&lt;br /&gt;
:; AVAIL&lt;br /&gt;
:: state/availability of the partition: see [[#Partition availability|below]]&lt;br /&gt;
&lt;br /&gt;
:; STATE&lt;br /&gt;
:: state (using [https://slurm.schedmd.com/sinfo.html#SECTION_NODE-STATE-CODES these codes])&lt;br /&gt;
:: typical values are &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;mixed&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that some of the resources are busy executing jobs while other are idle, and &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;allocated&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that all of the resources are in use&lt;br /&gt;
&lt;br /&gt;
:; DEFAULTTIME&lt;br /&gt;
:: default runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:; TIMELIMIT&lt;br /&gt;
:: maximum runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
The asterisk at the end of the partition name indicates the default partition, i.e. the one on which jobs which do not ask for a specific partition are run.&lt;br /&gt;
&lt;br /&gt;
Command &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; does not tell you about the &amp;#039;&amp;#039;jobs&amp;#039;&amp;#039; submitted to a partition. This information is obtained, instead, with [[User Jobs#Inspecting jobs with squeue|command &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;]].&lt;br /&gt;
&lt;br /&gt;
== Partition availability ==&lt;br /&gt;
&lt;br /&gt;
The most important information that &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; provides is the &amp;#039;&amp;#039;&amp;#039;availability&amp;#039;&amp;#039;&amp;#039; (also called &amp;#039;&amp;#039;&amp;#039;state&amp;#039;&amp;#039;&amp;#039;) of partitions. This is shown in column &amp;quot;AVAIL&amp;quot;. Possible partition states are:&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is available&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed as soon as resources allow&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is in the process of becoming unavailable (i.e., of entering the &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; state: see below)&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is unavailable&lt;br /&gt;
:: There are no running jobs&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
When a partition goes from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; no harm is done to running jobs. In a normally functioning SLURM system, the passage from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; happens only when no jobs are running on the partition. If (e.g., due to a malfunction) the passage happens with jobs still running, they get killed.&lt;br /&gt;
&lt;br /&gt;
A partition in state &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; requires intervention by a [[Roles|Job Administrator]] to be restored to &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
== Default values ==&lt;br /&gt;
&lt;br /&gt;
The features of SLURM partitions, including the &amp;#039;&amp;#039;&amp;#039;default values&amp;#039;&amp;#039;&amp;#039; which are applied to jobs that do not make explicit requests, can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scontrol show partition&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PartitionName=jobs&lt;br /&gt;
   AllowGroups=ALL AllowAccounts=ALL AllowQos=nogpu,gpulight,gpu,gpuwide,gpuheavy-20,gpuheavy-40&lt;br /&gt;
   AllocNodes=ALL Default=YES QoS=N/A&lt;br /&gt;
=&amp;gt; DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO&lt;br /&gt;
   MaxNodes=UNLIMITED MaxTime=3-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED&lt;br /&gt;
   Nodes=gn01&lt;br /&gt;
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO&lt;br /&gt;
   OverTimeLimit=NONE PreemptMode=OFF&lt;br /&gt;
   State=UP TotalCPUs=48 TotalNodes=1 SelectTypeParameters=NONE&lt;br /&gt;
   JobDefaults=(null)&lt;br /&gt;
=&amp;gt; DefMemPerNode=4096 MaxMemPerNode=UNLIMITED&lt;br /&gt;
   TRES=cpu=48,mem=1011435M,node=1,billing=49,gres/gpu=13,gres/gpu:3g.20gb=5,gres/gpu:40gb=3,gres/gpu:4g.20gb=5&lt;br /&gt;
   TRESBillingWeights=cpu=1.0,gres/gpu:3g.20gb=6.0,gres/gpu:4g.20gb=6.0,gres/gpu:40gb=6.0,mem=0.05g&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In the example, we have highlighted with  &amp;#039;&amp;#039;&amp;#039;=&amp;gt;&amp;#039;&amp;#039;&amp;#039;  the most relevant default values for Mufasa users, i.e.:&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefaultTime&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default execution time assigned to a job run on the partition (e.g., 1 hour)&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefMemPerNode&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default amount of RAM assigned to a job run on the partition (e.g., 4GB)&lt;br /&gt;
&lt;br /&gt;
= System resources subjected to limitations =&lt;br /&gt;
&lt;br /&gt;
In systems based on SLURM like Mufasa, &amp;#039;&amp;#039;&amp;#039;TRES (Trackable RESources)&amp;#039;&amp;#039;&amp;#039; are (from [https://slurm.schedmd.com/tres.html SLURM&amp;#039;s documentation] &amp;quot;&amp;#039;&amp;#039;resources that can be tracked for usage or used to enforce limits against.&amp;#039;&amp;#039;&amp;quot;&lt;br /&gt;
&lt;br /&gt;
TRES include CPUs, RAM and &amp;#039;&amp;#039;&amp;#039;GRES&amp;#039;&amp;#039;&amp;#039;. The last term stands for &amp;#039;&amp;#039;Generic RESources&amp;#039;&amp;#039; that a job may need for its execution. In Mufasa, the only &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; resources are the GPUs.&lt;br /&gt;
&lt;br /&gt;
== &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax ==&lt;br /&gt;
&lt;br /&gt;
To ask SLURM to assign GRES resources (i.e., GPUs) to a job, a special syntax must be used. Precisely, the name of each GPU resource takes the form&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:Name:Type&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
Considering the [[System#CPUs and GPUs|GPU complement of Mufasa]], &amp;lt;code&amp;gt;Type&amp;lt;/code&amp;gt; takes the following values:&lt;br /&gt;
&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:40gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 40 Gbytes of RAM&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:4g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 4 compute units&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 3 compute units&lt;br /&gt;
&lt;br /&gt;
So, for instance,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
identifies a resource corresponding to a GPU with 20 GB of RAM and 3 compute units.&lt;br /&gt;
&lt;br /&gt;
When asking for a GRES resource (e.g., in an &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command or an &amp;lt;code&amp;gt;SBATCH&amp;lt;/code&amp;gt; directive of an [[User Jobs#Using execution scripts to run jobs|execution script]]), the syntax required by SLURM is&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:&amp;lt;Type&amp;gt;:&amp;lt;Quantity&amp;gt;&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
where &amp;lt;code&amp;gt;Quantity&amp;lt;/code&amp;gt; is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of type &amp;lt;code&amp;gt;4g.20gb&amp;lt;/code&amp;gt; the syntax is&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:4g.20gb:2&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SLURM&amp;#039;s &amp;#039;&amp;#039;generic resources&amp;#039;&amp;#039; are defined in &amp;lt;code&amp;gt;/etc/slurm/gres.conf&amp;lt;/code&amp;gt;. In order to make GPUs available to SLURM&amp;#039;s &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; management, Mufasa makes use of Nvidia&amp;#039;s [https://developer.nvidia.com/nvidia-management-library-nvml NVML library]. For additional information see [https://slurm.schedmd.com/gres.html SLURM&amp;#039;s documentation].&lt;/div&gt;</summary>
		<author><name>GiulioFontana</name></author>
	</entry>
	<entry>
		<id>https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2577</id>
		<title>SLURM</title>
		<link rel="alternate" type="text/html" href="https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2577"/>
		<updated>2026-11-03T14:41:18Z</updated>

		<summary type="html">&lt;p&gt;GiulioFontana: /* Elements determining job priority */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page presents the features of SLURM that are most relevant to Mufasa&amp;#039;s [[Roles|Job Users]]. Job Users can submit jobs for execution, cancel their own jobs, and see other users&amp;#039; jobs (but not intervene on them).&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa &amp;#039;&amp;#039;&amp;#039;must use SLURM&amp;#039;&amp;#039;&amp;#039; to run resource-heavy processes, i.e. computing jobs that require one or more of the following:&lt;br /&gt;
* GPUs&lt;br /&gt;
* multiple CPUs&lt;br /&gt;
* powerful CPUs&lt;br /&gt;
* a significant amount of RAM&lt;br /&gt;
&lt;br /&gt;
In fact, only processes run via SLURM have access to all the resources of Mufasa. Processes run outside SLURM are executed by the [[System#Login server|login server]] virtual machine, which has minimal resources and no GPUs. Using SLURM is therefore the only way to execute resource-heavy jobs on Mufasa (this is a key difference between Mufasa 1.0 and Mufasa 2.0).&lt;br /&gt;
&lt;br /&gt;
= SLURM in a nutshell =&lt;br /&gt;
&lt;br /&gt;
Computation jobs on Mufasa needs to be launched via [[System#The SLURM job scheduling system|SLURM]]. SLURM provides jobs with access to the [[#System resources subjected to limitations|physical resources]] of Mufasa, such as CPUs, GPUs and RAM. Thanks to SLURM, processing jobs share system resources, optimising their occupation and availability. &lt;br /&gt;
&lt;br /&gt;
When a user runs a job, the job does not get executed immediately and is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039;. SLURM executes jobs according to their order in the queue: the top job in the queue gets executed as soon as the necessary resources are available, while jobs lower in the queue wait longer. The position of a job in the queue is due to the &amp;#039;&amp;#039;&amp;#039;[[#Job priority|priority]]&amp;#039;&amp;#039;&amp;#039; assigned to it by SLURM, with higher-priority jobs closer to the top. As a general rule:&lt;br /&gt;
&lt;br /&gt;
;: &amp;#039;&amp;#039;&amp;#039;the greater the fraction of Mufasa&amp;#039;s overall resources that a job asks for, the lower the job&amp;#039;s priority will be&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
The priority mechanism is used to encourage users to use Mufasa&amp;#039;s resources (i.e.: GPUs, CPUs, RAM, execution time) in an effective and equitable manner. This page includes a [[#How_to_maximise_the_priority_of_your_jobs|chart explaining how to maximise the priority of your jobs]].&lt;br /&gt;
&lt;br /&gt;
The &amp;#039;&amp;#039;&amp;#039;time&amp;#039;&amp;#039;&amp;#039; available to a job for its execution is controlled by SLURM. When a user requests execution of a job, they must specify the duration of the time slot that the job needs. The job must complete its execution before the end of the requested time slot, because at the end of such slot it gets killed by SLURM.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0 access to system resources is managed via SLURM&amp;#039;s [[#SLURM Quality of Service (QOS)|Quality of Service (QOS)]] mechanism (Mufasa 1.0 used [[#SLURM_partitions|partitions]] instead). While [[User Jobs#Running jobs with SLURM|launching a processing job via SLURM]], the user must always specify the chosen QOS. QOSes differ in the set of resources that they provide access to because each of them is designed to fit a given type of job.&lt;br /&gt;
&lt;br /&gt;
Mufasa sets limits to the number of jobs by the same user. This page includes a [[#Limits on jobs by the same user|table summarising such limits]].&lt;br /&gt;
&lt;br /&gt;
= SLURM Quality of Service (QOS) =&lt;br /&gt;
&lt;br /&gt;
Through &amp;#039;&amp;#039;&amp;#039;Quality of Services&amp;#039;&amp;#039;&amp;#039; (&amp;#039;&amp;#039;&amp;#039;QOSes&amp;#039;&amp;#039;&amp;#039;), SLURM lets system configurators assign a name to a set of related constraints.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, QOSes are used to define different levels of access to the server&amp;#039;s resources. When [[User Jobs|executing a job with SLURM]], a user must always &amp;#039;&amp;#039;&amp;#039;specify the QOS&amp;#039;&amp;#039;&amp;#039; that their job will use: this choice, in turn, determines what resources the job is able to access and influences the [[#Job priority|priority]] of the job.&lt;br /&gt;
&lt;br /&gt;
Mufasa&amp;#039;s QOSes and their features can be inspected with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list qos format=name%-11,priority,MaxSubmit,maxwall,maxtres%-80,mintres%-18&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following (see below for the same data in table form):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
Name          Priority MaxSubmit     MaxWall MaxTRES                                                                          MinTRES            &lt;br /&gt;
----------- ---------- --------- ----------- -------------------------------------------------------------------------------- ------------------ &lt;br /&gt;
normal               0                                                                                                                           &lt;br /&gt;
nogpu                4         1  3-00:00:00 cpu=16,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=128G                    &lt;br /&gt;
gpuheavy-20          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=2,mem=128G             gres/gpu:4g.20gb=1 &lt;br /&gt;
gpuheavy-40          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=1,gres/gpu:4g.20gb=0,mem=128G             gres/gpu:40gb=1    &lt;br /&gt;
gpulight             8         1    12:00:00 cpu=2,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpu                  2         1  1-00:00:00 cpu=8,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpuwide              2         2  1-00:00:00 cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=1,mem=64G              gres/gpu:4g.20gb=1 &lt;br /&gt;
build               32         1    02:00:00 cpu=2,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=16G&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The columns of this output are the following:&lt;br /&gt;
&lt;br /&gt;
:; Name&lt;br /&gt;
:: name of the QOS&lt;br /&gt;
&lt;br /&gt;
:; Priority&lt;br /&gt;
:: priority tier associated to the QOS (higher value = higher priority): see [[#Job priority|Job priority]] for details&lt;br /&gt;
&lt;br /&gt;
:; MaxSubmit&lt;br /&gt;
:: maximum number of jobs from a single user that can be submitted to SLURM with this QOS; submitted jobs include both running and queued jobs&lt;br /&gt;
:: See [[#Limits on jobs by the same user|Limits on jobs by the same user]] for an overview of the limits on jobs set by Mufasa.&lt;br /&gt;
&lt;br /&gt;
:; MaxWall&lt;br /&gt;
:: maximum wall clock duration of the jobs using the QOS (after which they are killed by SLURM), in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
:: For some QOSes these are not set: it means that they are determined by the [[#SLURM partitions|partition]]. Partitions also define the [[#Default values|default duration]] of jobs.&lt;br /&gt;
&lt;br /&gt;
:; MaxTRES&lt;br /&gt;
:: amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) available to a job using the QOS, where&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;cpu=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of CPUs (i.e., processor cores) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of CPUs specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;gres/&amp;#039;&amp;#039;gpu:Type&amp;#039;&amp;#039;=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of GPUs of class &amp;lt;code&amp;gt;&amp;#039;&amp;#039;Type&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; (see [[User Jobs#gres syntax|&amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax]]) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; (for QOSes that allow access to GPUs) if not specified, the job cannot be launched&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;mem=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;G&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum amount of system RAM is &amp;#039;&amp;#039;K&amp;#039;&amp;#039; GBytes&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of RAM specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
&lt;br /&gt;
:; MinTRES&lt;br /&gt;
:: minimum amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) that a job using the QOS must request in order to actually get executed by SLURM.&lt;br /&gt;
:: (If your job does not actually need these resources, you&amp;#039;ve chosen the wrong QOS: you can use one with a higher priority.)&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; QOS is the default one, and exists only to ensure that users always specify a QOS when running a job. Since &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; has zero priority and no resources, a job run using this QOS would never be run.&lt;br /&gt;
&lt;br /&gt;
The information provided by the &amp;lt;code&amp;gt;sacctmgr list qos&amp;lt;/code&amp;gt; command above is summarised by the following table:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! QOS&lt;br /&gt;
! Priority&amp;lt;/br&amp;gt;tier&lt;br /&gt;
! Max&amp;lt;/br&amp;gt;Submit&lt;br /&gt;
! MaxWall&amp;lt;/br&amp;gt;[h]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;CPUs&lt;br /&gt;
! max RAM&amp;lt;/br&amp;gt;[GB]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;3g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;4g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;40GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | build&lt;br /&gt;
| 32&lt;br /&gt;
| 1&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 16&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpulight&lt;br /&gt;
| 8&lt;br /&gt;
| 1&lt;br /&gt;
| 12&lt;br /&gt;
| 2&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | nogpu&lt;br /&gt;
| 4&lt;br /&gt;
| 1&lt;br /&gt;
| 72&lt;br /&gt;
| 16&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpu&lt;br /&gt;
| 2&lt;br /&gt;
| 1&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuwide&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-20&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| 2 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-40&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
A key piece of information in the table above is the &amp;#039;&amp;#039;&amp;#039;priority tier&amp;#039;&amp;#039;&amp;#039; associated to each QOS. In Mufasa 2.0, priority tiers are used to encourage users to use the &amp;#039;&amp;#039;&amp;#039;least powerful QOS that is compatible with their needs&amp;#039;&amp;#039;&amp;#039;, where &amp;quot;powerful&amp;quot; means &amp;quot;rich with resources&amp;quot;. Less powerful QOSes increase the priority of the jobs that use them, so these jobs tend to be executed sooner. See [[#Job priority|Job priority]] for details about how priority affects the execution order of jobs in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
== The &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS ==&lt;br /&gt;
&lt;br /&gt;
This QOS is specifically designed to be used by Mufasa users to &amp;#039;&amp;#039;&amp;#039;quickly build [[System#Containers|container images]]&amp;#039;&amp;#039;&amp;#039;. Its associated priority tier is very high: this, combined with the fact that jobs using this QOS require few resources, means that such jobs usually get executed very soon.&lt;br /&gt;
&lt;br /&gt;
On the other hand, the limited resources and the lack of access to the GPUs make the &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS unsuitable for other tasks.&lt;br /&gt;
&lt;br /&gt;
See [[Singularity#Building Singularity images|Building Singularity images]] for directions about building Singularity container images.&lt;br /&gt;
&lt;br /&gt;
== Restricted QOSes ==&lt;br /&gt;
&lt;br /&gt;
In Mufasa, the most powerful QOSes are reserved to researchers (called &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users: these include academic personnel and Ph.D. students). M.Sc. students, called &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users, cannot use them while running jobs.&lt;br /&gt;
&lt;br /&gt;
See [[#research users and students users|below]] to understand how &amp;#039;&amp;#039;user categories&amp;#039;&amp;#039; work in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
= &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users =&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa belong to two &amp;#039;&amp;#039;&amp;#039;user categories&amp;#039;&amp;#039;&amp;#039;. Categories provide users with different access to  to Mufasa&amp;#039;s resources: the idea being to provide researchers with more access while still letting students benefit from the server&amp;#039;s capabilities.&lt;br /&gt;
&lt;br /&gt;
User categories in Mufasa are the following:&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. academic personnel and Ph.D. students&lt;br /&gt;
::: * can use all QOSes, including the the [[#Restricted QOSes|restricted ones]]&lt;br /&gt;
::: * their jobs have a higher &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a higher number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. M.Sc. students&lt;br /&gt;
::: * cannot use the [[#Restricted QOSes|restricted QOSes]]&lt;br /&gt;
::: * their jobs have a lower &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a lower number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
You can inspect the differences in priority and running jobs between &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association format=&amp;quot;account,priority,maxjobs&amp;quot; | grep -E &amp;#039;Account|research|students&amp;#039;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
   Account   Priority MaxJobs &lt;br /&gt;
  research          4       2 &lt;br /&gt;
  students          1       1&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
To know what limits apply to your own user, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association where user=$USER format=&amp;quot;user,priority,maxjobs,qos%-60&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
      User   Priority MaxJobs QOS                                                          &lt;br /&gt;
---------- ---------- ------- ------------------------------------------------------------ &lt;br /&gt;
    preali          4       2 build,gpu,gpuheavy-20,gpuheavy-40,gpulight,gpuwide,nogpu&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The list under &amp;quot;QOS&amp;quot; shows what QOSes your user is allowed to use when [[User Jobs|running jobs]]. &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users can use all of them, while &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users can only access a subset of them.&lt;br /&gt;
&lt;br /&gt;
= Limits on jobs by the same user =&lt;br /&gt;
&lt;br /&gt;
Mufasa enforces limits on the number of jobs from a single user. Such limits aim at preventing users from &amp;quot;hogging&amp;quot; system resources, and apply to:&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;submitted jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that the user asked SLURM to execute, each of which may currently be either running or queued&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;running jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that are currently in execution&lt;br /&gt;
&lt;br /&gt;
Limits on submitted jobs are set via [[#SLURM Quality of Service (QOS)|QOSes]], while limits on running jobs are set via [[#research users and students users|user category]].&lt;br /&gt;
&lt;br /&gt;
The following table summarises what limits exist:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! &lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;running&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;submitted&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | global limits&amp;lt;br/&amp;gt;(system-wide)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to limits set by individual QOSes (below)&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | limits for&amp;lt;br/&amp;gt;individual&amp;lt;/br&amp;gt;QOSes&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to global limits (above)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for the&amp;#039;&amp;#039; &amp;lt;code&amp;gt;gpuwide&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;QOS&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for each of the other QOSes&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
= Job priority =&lt;br /&gt;
&lt;br /&gt;
Once the execution of a job has been requested, the job is not run immediately: it is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039; by SLURM, together with all the other jobs awaiting execution. The job on top of the queue at any time is the first to be put into execution as soon as the resources it requires are available. &lt;br /&gt;
&lt;br /&gt;
The order of the items in the job queue depends on their &amp;#039;&amp;#039;&amp;#039;priority&amp;#039;&amp;#039;&amp;#039;. The job with the highest priority is put by SLURM on top of the queue, while all other jobs are put in the queue in order of descending priority.&lt;br /&gt;
&lt;br /&gt;
The goal of SLURM is to maximise resource availability: i.e., to ensure the shorter possible wait time before job execution. To achieve this goal, in Mufasa SLURM is configured to &amp;#039;&amp;#039;&amp;#039;encourage users not to ask for resources or execution time that their job doesn&amp;#039;t need&amp;#039;&amp;#039;&amp;#039;. This is done via the priority mechanism: the more resources and/or time a job requests, the lower its priority will be; and the later it will be executed.&lt;br /&gt;
&lt;br /&gt;
Priority management in Mufasa is designed to set up a &amp;#039;&amp;#039;&amp;#039;virtuous cycle&amp;#039;&amp;#039;&amp;#039; where users, by carefully choosing what to ask for, obtain two results:&lt;br /&gt;
* they ensure that their job is executed as soon as possible;&lt;br /&gt;
* they leave as much as possible of Mufasa&amp;#039;s resources free for other users&amp;#039;s jobs.&lt;br /&gt;
&lt;br /&gt;
== Elements determining job priority ==&lt;br /&gt;
In Mufasa, the priority of a job is computed by SLURM according to the following elements:&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#research users and students users|User category]]&amp;#039;&amp;#039;&amp;#039; (i.e., &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;)&lt;br /&gt;
::: Used to provide higher priority to jobs run by &amp;#039;&amp;#039;&amp;#039;research personnel&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#SLURM Quality of Service (QOS)|QOS]]&amp;#039;&amp;#039;&amp;#039; used by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;less resources&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Number of CPUs&amp;#039;&amp;#039;&amp;#039; requested by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;a lower number of CPUs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job Age&amp;#039;&amp;#039;&amp;#039;, i.e. the time that the job has been waiting in the queue&lt;br /&gt;
::: Used to provide higher priority to jobs which have been &amp;#039;&amp;#039;&amp;#039;queued for a longer time&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;FairShare&amp;#039;&amp;#039;&amp;#039;, i.e. a factor computed by SLURM to balance use of the system by different users&lt;br /&gt;
::: Used to provide higher priority to jobs by users who &amp;#039;&amp;#039;&amp;#039;used Mufasa&amp;#039;s resources less than others&amp;#039;&amp;#039;&amp;#039; (see [[#How FairShare works|below]])&lt;br /&gt;
&lt;br /&gt;
You can understand how each of these elements contribute to the priority of the jobs waiting for execution (both yours and of other users) with command &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;. A tailored version of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; that only displays the parameters listed above is the following:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sprio -o &amp;quot;%6i %9u %.5Y %.10B %.5Q %.9T %.5A %.5F&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Important note!&lt;br /&gt;
:: The output of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; is only relevant for jobs &amp;#039;&amp;#039;&amp;#039;not yet in execution&amp;#039;&amp;#039;&amp;#039;. For an already running job, the value shown by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; is the priority that the job would get if still queued, which usually does not correspond to its actual priority.&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Here is an example of its output:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
JOBID  USER      PRIOR      ASSOC   QOS      TRES   AGE FAIRS&lt;br /&gt;
5774   itombini   3492       2500   750  cpu=-800   906   136&lt;br /&gt;
5780   fbottazzi  2266        625   750  cpu=-600   420  1071&lt;br /&gt;
5783   mzoccola   1766        625   750  cpu=-800   217   974&lt;br /&gt;
5790   amoglia    2717       2500   375  cpu=-800   156   487&lt;br /&gt;
5791   qliu       2232       2500   375  cpu=-800   138    19&lt;br /&gt;
5792   fpastori   2197       2500   375  cpu=-800    84    39&lt;br /&gt;
5793   bagostini  2034        625  1500 cpu=-1200    18  1091&lt;br /&gt;
5794   lpassoni   2517       2500   750  cpu=-800     9    58&lt;br /&gt;
5795   lpassoni   2517       2500   750  cpu=-800     9    58&lt;br /&gt;
5796   adurat     3101       2500   750  cpu=-800     8   643&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
In it, column ASSOC (i.e., association) corresponds to user category, while column TRES corresponds to the number of CPUs. Note that the contribution of CPUs to priority is &amp;#039;&amp;#039;negative&amp;#039;&amp;#039;, i.e. it is a penalty. Column PRIOR provides the overall priority of each job.&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
=== How FairShare works ===&lt;br /&gt;
In Mufasa 2.0, the FairShare of each user is computed according to these rules:&lt;br /&gt;
# It considers the quantity used by (current and past) user jobs of:&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;CPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of CPUs used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;RAM&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the amount of RAM used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;GPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of GPUs used&lt;br /&gt;
#** note: 40 GB GPUs have double the impact on FairShare than 20 GB GPUs&lt;br /&gt;
# The impact on FairShare of any use of resources is &amp;#039;&amp;#039;&amp;#039;proportional to its duration&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
#* example: using 32 GB of RAM for 48h has the same impact of using 64 GB of RAM for 24h&lt;br /&gt;
# FairShare has a &amp;quot;fading memory&amp;quot;: i.e., resource use has less and less impact on FairShare the farther in the past it is&lt;br /&gt;
#* consequence: over time, the &amp;quot;history&amp;quot; of a user gets forgotten by FairShare&lt;br /&gt;
&lt;br /&gt;
== How to maximise the priority of your jobs ==&lt;br /&gt;
&lt;br /&gt;
Every time you run a SLURM job, follow these guidelines:&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Choose the less powerful QOS compatible with the needs of your job&lt;br /&gt;
:: QOSes with access to less resources have higher base priority&lt;br /&gt;
&lt;br /&gt;
; Only request CPUs and RAM that your job really needs&lt;br /&gt;
:: Usually, code exploits multiple CPUs only if specifically designed to do so: if unsure, only ask for 1 CPU&lt;br /&gt;
:: The fewer resources your job asks for, the less it will wait before they become available&lt;br /&gt;
:: Asking for fewer CPUs and/or less RAM improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Do not request more time than your jobs needs to complete&lt;br /&gt;
:: Make a worst-case estimate and only ask for that duration&lt;br /&gt;
:: Asking for less of Mufasa&amp;#039;s time improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Debug and test code using less powerful QOSes before running it with more powerful QOSes&lt;br /&gt;
:: Your test jobs will get a higher priority and your FairShare will improve&lt;br /&gt;
&lt;br /&gt;
; Cancel jobs that you don&amp;#039;t need them anymore&lt;br /&gt;
:: Always use [[User_Jobs#Cancelling_a_job_with_scancel|scancel]] to delete completed (or crashed) jobs: your Fairshare will improve&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Protip: it&amp;#039;s a good idea to [[User Jobs#Looking for unused GPUs|check for unused GPUs]] before choosing what to request. Requesting a GPU that is currently idle will help your job get executed sooner.&lt;br /&gt;
&lt;br /&gt;
= SLURM partitions =&lt;br /&gt;
&lt;br /&gt;
Partitions are another mechanism provided by SLURM to create different levels of access to system resources. Since in Mufasa 2.0 access to resources is controlled via [[#SLURM Quality of Service (QOS)|QOSes]], partitions are not very relevant. (This is one of the main differences between Mufasa 1.0 and Mufasa 2.0.)&lt;br /&gt;
&lt;br /&gt;
Note, however, that the default values for some features of SLURM jobs (e.g., duration) are [[#Default values|set by the partition]].&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, there is a single SLURM partition, called &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt;, and all jobs run on it. The state of &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt; can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sinfo -o &amp;quot;%10P %5a %9T %11L %10l&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PARTITION  AVAIL STATE     DEFAULTTIME TIMELIMIT &lt;br /&gt;
jobs*      up    idle      1:00:00     3-00:00:00&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where columns correspond to the following information:&lt;br /&gt;
&lt;br /&gt;
:; PARTITION&lt;br /&gt;
:: name of the partition; the asterisks indicates that it&amp;#039;s the default one&lt;br /&gt;
&lt;br /&gt;
:; AVAIL&lt;br /&gt;
:: state/availability of the partition: see [[#Partition availability|below]]&lt;br /&gt;
&lt;br /&gt;
:; STATE&lt;br /&gt;
:: state (using [https://slurm.schedmd.com/sinfo.html#SECTION_NODE-STATE-CODES these codes])&lt;br /&gt;
:: typical values are &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;mixed&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that some of the resources are busy executing jobs while other are idle, and &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;allocated&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that all of the resources are in use&lt;br /&gt;
&lt;br /&gt;
:; DEFAULTTIME&lt;br /&gt;
:: default runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:; TIMELIMIT&lt;br /&gt;
:: maximum runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
The asterisk at the end of the partition name indicates the default partition, i.e. the one on which jobs which do not ask for a specific partition are run.&lt;br /&gt;
&lt;br /&gt;
Command &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; does not tell you about the &amp;#039;&amp;#039;jobs&amp;#039;&amp;#039; submitted to a partition. This information is obtained, instead, with [[User Jobs#Inspecting jobs with squeue|command &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;]].&lt;br /&gt;
&lt;br /&gt;
== Partition availability ==&lt;br /&gt;
&lt;br /&gt;
The most important information that &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; provides is the &amp;#039;&amp;#039;&amp;#039;availability&amp;#039;&amp;#039;&amp;#039; (also called &amp;#039;&amp;#039;&amp;#039;state&amp;#039;&amp;#039;&amp;#039;) of partitions. This is shown in column &amp;quot;AVAIL&amp;quot;. Possible partition states are:&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is available&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed as soon as resources allow&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is in the process of becoming unavailable (i.e., of entering the &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; state: see below)&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is unavailable&lt;br /&gt;
:: There are no running jobs&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
When a partition goes from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; no harm is done to running jobs. In a normally functioning SLURM system, the passage from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; happens only when no jobs are running on the partition. If (e.g., due to a malfunction) the passage happens with jobs still running, they get killed.&lt;br /&gt;
&lt;br /&gt;
A partition in state &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; requires intervention by a [[Roles|Job Administrator]] to be restored to &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
== Default values ==&lt;br /&gt;
&lt;br /&gt;
The features of SLURM partitions, including the &amp;#039;&amp;#039;&amp;#039;default values&amp;#039;&amp;#039;&amp;#039; which are applied to jobs that do not make explicit requests, can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scontrol show partition&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PartitionName=jobs&lt;br /&gt;
   AllowGroups=ALL AllowAccounts=ALL AllowQos=nogpu,gpulight,gpu,gpuwide,gpuheavy-20,gpuheavy-40&lt;br /&gt;
   AllocNodes=ALL Default=YES QoS=N/A&lt;br /&gt;
=&amp;gt; DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO&lt;br /&gt;
   MaxNodes=UNLIMITED MaxTime=3-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED&lt;br /&gt;
   Nodes=gn01&lt;br /&gt;
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO&lt;br /&gt;
   OverTimeLimit=NONE PreemptMode=OFF&lt;br /&gt;
   State=UP TotalCPUs=48 TotalNodes=1 SelectTypeParameters=NONE&lt;br /&gt;
   JobDefaults=(null)&lt;br /&gt;
=&amp;gt; DefMemPerNode=4096 MaxMemPerNode=UNLIMITED&lt;br /&gt;
   TRES=cpu=48,mem=1011435M,node=1,billing=49,gres/gpu=13,gres/gpu:3g.20gb=5,gres/gpu:40gb=3,gres/gpu:4g.20gb=5&lt;br /&gt;
   TRESBillingWeights=cpu=1.0,gres/gpu:3g.20gb=6.0,gres/gpu:4g.20gb=6.0,gres/gpu:40gb=6.0,mem=0.05g&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In the example, we have highlighted with  &amp;#039;&amp;#039;&amp;#039;=&amp;gt;&amp;#039;&amp;#039;&amp;#039;  the most relevant default values for Mufasa users, i.e.:&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefaultTime&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default execution time assigned to a job run on the partition (e.g., 1 hour)&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefMemPerNode&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default amount of RAM assigned to a job run on the partition (e.g., 4GB)&lt;br /&gt;
&lt;br /&gt;
= System resources subjected to limitations =&lt;br /&gt;
&lt;br /&gt;
In systems based on SLURM like Mufasa, &amp;#039;&amp;#039;&amp;#039;TRES (Trackable RESources)&amp;#039;&amp;#039;&amp;#039; are (from [https://slurm.schedmd.com/tres.html SLURM&amp;#039;s documentation] &amp;quot;&amp;#039;&amp;#039;resources that can be tracked for usage or used to enforce limits against.&amp;#039;&amp;#039;&amp;quot;&lt;br /&gt;
&lt;br /&gt;
TRES include CPUs, RAM and &amp;#039;&amp;#039;&amp;#039;GRES&amp;#039;&amp;#039;&amp;#039;. The last term stands for &amp;#039;&amp;#039;Generic RESources&amp;#039;&amp;#039; that a job may need for its execution. In Mufasa, the only &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; resources are the GPUs.&lt;br /&gt;
&lt;br /&gt;
== &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax ==&lt;br /&gt;
&lt;br /&gt;
To ask SLURM to assign GRES resources (i.e., GPUs) to a job, a special syntax must be used. Precisely, the name of each GPU resource takes the form&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:Name:Type&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
Considering the [[System#CPUs and GPUs|GPU complement of Mufasa]], &amp;lt;code&amp;gt;Type&amp;lt;/code&amp;gt; takes the following values:&lt;br /&gt;
&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:40gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 40 Gbytes of RAM&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:4g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 4 compute units&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 3 compute units&lt;br /&gt;
&lt;br /&gt;
So, for instance,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
identifies a resource corresponding to a GPU with 20 GB of RAM and 3 compute units.&lt;br /&gt;
&lt;br /&gt;
When asking for a GRES resource (e.g., in an &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command or an &amp;lt;code&amp;gt;SBATCH&amp;lt;/code&amp;gt; directive of an [[User Jobs#Using execution scripts to run jobs|execution script]]), the syntax required by SLURM is&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:&amp;lt;Type&amp;gt;:&amp;lt;Quantity&amp;gt;&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
where &amp;lt;code&amp;gt;Quantity&amp;lt;/code&amp;gt; is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of type &amp;lt;code&amp;gt;4g.20gb&amp;lt;/code&amp;gt; the syntax is&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:4g.20gb:2&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SLURM&amp;#039;s &amp;#039;&amp;#039;generic resources&amp;#039;&amp;#039; are defined in &amp;lt;code&amp;gt;/etc/slurm/gres.conf&amp;lt;/code&amp;gt;. In order to make GPUs available to SLURM&amp;#039;s &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; management, Mufasa makes use of Nvidia&amp;#039;s [https://developer.nvidia.com/nvidia-management-library-nvml NVML library]. For additional information see [https://slurm.schedmd.com/gres.html SLURM&amp;#039;s documentation].&lt;/div&gt;</summary>
		<author><name>GiulioFontana</name></author>
	</entry>
	<entry>
		<id>https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2576</id>
		<title>SLURM</title>
		<link rel="alternate" type="text/html" href="https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2576"/>
		<updated>2026-11-03T14:41:00Z</updated>

		<summary type="html">&lt;p&gt;GiulioFontana: /* Elements determining job priority */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page presents the features of SLURM that are most relevant to Mufasa&amp;#039;s [[Roles|Job Users]]. Job Users can submit jobs for execution, cancel their own jobs, and see other users&amp;#039; jobs (but not intervene on them).&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa &amp;#039;&amp;#039;&amp;#039;must use SLURM&amp;#039;&amp;#039;&amp;#039; to run resource-heavy processes, i.e. computing jobs that require one or more of the following:&lt;br /&gt;
* GPUs&lt;br /&gt;
* multiple CPUs&lt;br /&gt;
* powerful CPUs&lt;br /&gt;
* a significant amount of RAM&lt;br /&gt;
&lt;br /&gt;
In fact, only processes run via SLURM have access to all the resources of Mufasa. Processes run outside SLURM are executed by the [[System#Login server|login server]] virtual machine, which has minimal resources and no GPUs. Using SLURM is therefore the only way to execute resource-heavy jobs on Mufasa (this is a key difference between Mufasa 1.0 and Mufasa 2.0).&lt;br /&gt;
&lt;br /&gt;
= SLURM in a nutshell =&lt;br /&gt;
&lt;br /&gt;
Computation jobs on Mufasa needs to be launched via [[System#The SLURM job scheduling system|SLURM]]. SLURM provides jobs with access to the [[#System resources subjected to limitations|physical resources]] of Mufasa, such as CPUs, GPUs and RAM. Thanks to SLURM, processing jobs share system resources, optimising their occupation and availability. &lt;br /&gt;
&lt;br /&gt;
When a user runs a job, the job does not get executed immediately and is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039;. SLURM executes jobs according to their order in the queue: the top job in the queue gets executed as soon as the necessary resources are available, while jobs lower in the queue wait longer. The position of a job in the queue is due to the &amp;#039;&amp;#039;&amp;#039;[[#Job priority|priority]]&amp;#039;&amp;#039;&amp;#039; assigned to it by SLURM, with higher-priority jobs closer to the top. As a general rule:&lt;br /&gt;
&lt;br /&gt;
;: &amp;#039;&amp;#039;&amp;#039;the greater the fraction of Mufasa&amp;#039;s overall resources that a job asks for, the lower the job&amp;#039;s priority will be&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
The priority mechanism is used to encourage users to use Mufasa&amp;#039;s resources (i.e.: GPUs, CPUs, RAM, execution time) in an effective and equitable manner. This page includes a [[#How_to_maximise_the_priority_of_your_jobs|chart explaining how to maximise the priority of your jobs]].&lt;br /&gt;
&lt;br /&gt;
The &amp;#039;&amp;#039;&amp;#039;time&amp;#039;&amp;#039;&amp;#039; available to a job for its execution is controlled by SLURM. When a user requests execution of a job, they must specify the duration of the time slot that the job needs. The job must complete its execution before the end of the requested time slot, because at the end of such slot it gets killed by SLURM.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0 access to system resources is managed via SLURM&amp;#039;s [[#SLURM Quality of Service (QOS)|Quality of Service (QOS)]] mechanism (Mufasa 1.0 used [[#SLURM_partitions|partitions]] instead). While [[User Jobs#Running jobs with SLURM|launching a processing job via SLURM]], the user must always specify the chosen QOS. QOSes differ in the set of resources that they provide access to because each of them is designed to fit a given type of job.&lt;br /&gt;
&lt;br /&gt;
Mufasa sets limits to the number of jobs by the same user. This page includes a [[#Limits on jobs by the same user|table summarising such limits]].&lt;br /&gt;
&lt;br /&gt;
= SLURM Quality of Service (QOS) =&lt;br /&gt;
&lt;br /&gt;
Through &amp;#039;&amp;#039;&amp;#039;Quality of Services&amp;#039;&amp;#039;&amp;#039; (&amp;#039;&amp;#039;&amp;#039;QOSes&amp;#039;&amp;#039;&amp;#039;), SLURM lets system configurators assign a name to a set of related constraints.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, QOSes are used to define different levels of access to the server&amp;#039;s resources. When [[User Jobs|executing a job with SLURM]], a user must always &amp;#039;&amp;#039;&amp;#039;specify the QOS&amp;#039;&amp;#039;&amp;#039; that their job will use: this choice, in turn, determines what resources the job is able to access and influences the [[#Job priority|priority]] of the job.&lt;br /&gt;
&lt;br /&gt;
Mufasa&amp;#039;s QOSes and their features can be inspected with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list qos format=name%-11,priority,MaxSubmit,maxwall,maxtres%-80,mintres%-18&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following (see below for the same data in table form):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
Name          Priority MaxSubmit     MaxWall MaxTRES                                                                          MinTRES            &lt;br /&gt;
----------- ---------- --------- ----------- -------------------------------------------------------------------------------- ------------------ &lt;br /&gt;
normal               0                                                                                                                           &lt;br /&gt;
nogpu                4         1  3-00:00:00 cpu=16,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=128G                    &lt;br /&gt;
gpuheavy-20          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=2,mem=128G             gres/gpu:4g.20gb=1 &lt;br /&gt;
gpuheavy-40          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=1,gres/gpu:4g.20gb=0,mem=128G             gres/gpu:40gb=1    &lt;br /&gt;
gpulight             8         1    12:00:00 cpu=2,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpu                  2         1  1-00:00:00 cpu=8,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpuwide              2         2  1-00:00:00 cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=1,mem=64G              gres/gpu:4g.20gb=1 &lt;br /&gt;
build               32         1    02:00:00 cpu=2,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=16G&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The columns of this output are the following:&lt;br /&gt;
&lt;br /&gt;
:; Name&lt;br /&gt;
:: name of the QOS&lt;br /&gt;
&lt;br /&gt;
:; Priority&lt;br /&gt;
:: priority tier associated to the QOS (higher value = higher priority): see [[#Job priority|Job priority]] for details&lt;br /&gt;
&lt;br /&gt;
:; MaxSubmit&lt;br /&gt;
:: maximum number of jobs from a single user that can be submitted to SLURM with this QOS; submitted jobs include both running and queued jobs&lt;br /&gt;
:: See [[#Limits on jobs by the same user|Limits on jobs by the same user]] for an overview of the limits on jobs set by Mufasa.&lt;br /&gt;
&lt;br /&gt;
:; MaxWall&lt;br /&gt;
:: maximum wall clock duration of the jobs using the QOS (after which they are killed by SLURM), in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
:: For some QOSes these are not set: it means that they are determined by the [[#SLURM partitions|partition]]. Partitions also define the [[#Default values|default duration]] of jobs.&lt;br /&gt;
&lt;br /&gt;
:; MaxTRES&lt;br /&gt;
:: amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) available to a job using the QOS, where&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;cpu=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of CPUs (i.e., processor cores) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of CPUs specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;gres/&amp;#039;&amp;#039;gpu:Type&amp;#039;&amp;#039;=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of GPUs of class &amp;lt;code&amp;gt;&amp;#039;&amp;#039;Type&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; (see [[User Jobs#gres syntax|&amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax]]) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; (for QOSes that allow access to GPUs) if not specified, the job cannot be launched&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;mem=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;G&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum amount of system RAM is &amp;#039;&amp;#039;K&amp;#039;&amp;#039; GBytes&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of RAM specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
&lt;br /&gt;
:; MinTRES&lt;br /&gt;
:: minimum amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) that a job using the QOS must request in order to actually get executed by SLURM.&lt;br /&gt;
:: (If your job does not actually need these resources, you&amp;#039;ve chosen the wrong QOS: you can use one with a higher priority.)&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; QOS is the default one, and exists only to ensure that users always specify a QOS when running a job. Since &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; has zero priority and no resources, a job run using this QOS would never be run.&lt;br /&gt;
&lt;br /&gt;
The information provided by the &amp;lt;code&amp;gt;sacctmgr list qos&amp;lt;/code&amp;gt; command above is summarised by the following table:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! QOS&lt;br /&gt;
! Priority&amp;lt;/br&amp;gt;tier&lt;br /&gt;
! Max&amp;lt;/br&amp;gt;Submit&lt;br /&gt;
! MaxWall&amp;lt;/br&amp;gt;[h]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;CPUs&lt;br /&gt;
! max RAM&amp;lt;/br&amp;gt;[GB]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;3g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;4g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;40GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | build&lt;br /&gt;
| 32&lt;br /&gt;
| 1&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 16&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpulight&lt;br /&gt;
| 8&lt;br /&gt;
| 1&lt;br /&gt;
| 12&lt;br /&gt;
| 2&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | nogpu&lt;br /&gt;
| 4&lt;br /&gt;
| 1&lt;br /&gt;
| 72&lt;br /&gt;
| 16&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpu&lt;br /&gt;
| 2&lt;br /&gt;
| 1&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuwide&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-20&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| 2 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-40&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
A key piece of information in the table above is the &amp;#039;&amp;#039;&amp;#039;priority tier&amp;#039;&amp;#039;&amp;#039; associated to each QOS. In Mufasa 2.0, priority tiers are used to encourage users to use the &amp;#039;&amp;#039;&amp;#039;least powerful QOS that is compatible with their needs&amp;#039;&amp;#039;&amp;#039;, where &amp;quot;powerful&amp;quot; means &amp;quot;rich with resources&amp;quot;. Less powerful QOSes increase the priority of the jobs that use them, so these jobs tend to be executed sooner. See [[#Job priority|Job priority]] for details about how priority affects the execution order of jobs in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
== The &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS ==&lt;br /&gt;
&lt;br /&gt;
This QOS is specifically designed to be used by Mufasa users to &amp;#039;&amp;#039;&amp;#039;quickly build [[System#Containers|container images]]&amp;#039;&amp;#039;&amp;#039;. Its associated priority tier is very high: this, combined with the fact that jobs using this QOS require few resources, means that such jobs usually get executed very soon.&lt;br /&gt;
&lt;br /&gt;
On the other hand, the limited resources and the lack of access to the GPUs make the &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS unsuitable for other tasks.&lt;br /&gt;
&lt;br /&gt;
See [[Singularity#Building Singularity images|Building Singularity images]] for directions about building Singularity container images.&lt;br /&gt;
&lt;br /&gt;
== Restricted QOSes ==&lt;br /&gt;
&lt;br /&gt;
In Mufasa, the most powerful QOSes are reserved to researchers (called &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users: these include academic personnel and Ph.D. students). M.Sc. students, called &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users, cannot use them while running jobs.&lt;br /&gt;
&lt;br /&gt;
See [[#research users and students users|below]] to understand how &amp;#039;&amp;#039;user categories&amp;#039;&amp;#039; work in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
= &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users =&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa belong to two &amp;#039;&amp;#039;&amp;#039;user categories&amp;#039;&amp;#039;&amp;#039;. Categories provide users with different access to  to Mufasa&amp;#039;s resources: the idea being to provide researchers with more access while still letting students benefit from the server&amp;#039;s capabilities.&lt;br /&gt;
&lt;br /&gt;
User categories in Mufasa are the following:&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. academic personnel and Ph.D. students&lt;br /&gt;
::: * can use all QOSes, including the the [[#Restricted QOSes|restricted ones]]&lt;br /&gt;
::: * their jobs have a higher &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a higher number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. M.Sc. students&lt;br /&gt;
::: * cannot use the [[#Restricted QOSes|restricted QOSes]]&lt;br /&gt;
::: * their jobs have a lower &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a lower number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
You can inspect the differences in priority and running jobs between &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association format=&amp;quot;account,priority,maxjobs&amp;quot; | grep -E &amp;#039;Account|research|students&amp;#039;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
   Account   Priority MaxJobs &lt;br /&gt;
  research          4       2 &lt;br /&gt;
  students          1       1&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
To know what limits apply to your own user, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association where user=$USER format=&amp;quot;user,priority,maxjobs,qos%-60&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
      User   Priority MaxJobs QOS                                                          &lt;br /&gt;
---------- ---------- ------- ------------------------------------------------------------ &lt;br /&gt;
    preali          4       2 build,gpu,gpuheavy-20,gpuheavy-40,gpulight,gpuwide,nogpu&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The list under &amp;quot;QOS&amp;quot; shows what QOSes your user is allowed to use when [[User Jobs|running jobs]]. &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users can use all of them, while &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users can only access a subset of them.&lt;br /&gt;
&lt;br /&gt;
= Limits on jobs by the same user =&lt;br /&gt;
&lt;br /&gt;
Mufasa enforces limits on the number of jobs from a single user. Such limits aim at preventing users from &amp;quot;hogging&amp;quot; system resources, and apply to:&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;submitted jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that the user asked SLURM to execute, each of which may currently be either running or queued&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;running jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that are currently in execution&lt;br /&gt;
&lt;br /&gt;
Limits on submitted jobs are set via [[#SLURM Quality of Service (QOS)|QOSes]], while limits on running jobs are set via [[#research users and students users|user category]].&lt;br /&gt;
&lt;br /&gt;
The following table summarises what limits exist:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! &lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;running&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;submitted&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | global limits&amp;lt;br/&amp;gt;(system-wide)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to limits set by individual QOSes (below)&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | limits for&amp;lt;br/&amp;gt;individual&amp;lt;/br&amp;gt;QOSes&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to global limits (above)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for the&amp;#039;&amp;#039; &amp;lt;code&amp;gt;gpuwide&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;QOS&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for each of the other QOSes&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
= Job priority =&lt;br /&gt;
&lt;br /&gt;
Once the execution of a job has been requested, the job is not run immediately: it is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039; by SLURM, together with all the other jobs awaiting execution. The job on top of the queue at any time is the first to be put into execution as soon as the resources it requires are available. &lt;br /&gt;
&lt;br /&gt;
The order of the items in the job queue depends on their &amp;#039;&amp;#039;&amp;#039;priority&amp;#039;&amp;#039;&amp;#039;. The job with the highest priority is put by SLURM on top of the queue, while all other jobs are put in the queue in order of descending priority.&lt;br /&gt;
&lt;br /&gt;
The goal of SLURM is to maximise resource availability: i.e., to ensure the shorter possible wait time before job execution. To achieve this goal, in Mufasa SLURM is configured to &amp;#039;&amp;#039;&amp;#039;encourage users not to ask for resources or execution time that their job doesn&amp;#039;t need&amp;#039;&amp;#039;&amp;#039;. This is done via the priority mechanism: the more resources and/or time a job requests, the lower its priority will be; and the later it will be executed.&lt;br /&gt;
&lt;br /&gt;
Priority management in Mufasa is designed to set up a &amp;#039;&amp;#039;&amp;#039;virtuous cycle&amp;#039;&amp;#039;&amp;#039; where users, by carefully choosing what to ask for, obtain two results:&lt;br /&gt;
* they ensure that their job is executed as soon as possible;&lt;br /&gt;
* they leave as much as possible of Mufasa&amp;#039;s resources free for other users&amp;#039;s jobs.&lt;br /&gt;
&lt;br /&gt;
== Elements determining job priority ==&lt;br /&gt;
In Mufasa, the priority of a job is computed by SLURM according to the following elements:&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#research users and students users|User category]]&amp;#039;&amp;#039;&amp;#039; (i.e., &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;)&lt;br /&gt;
::: Used to provide higher priority to jobs run by &amp;#039;&amp;#039;&amp;#039;research personnel&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#SLURM Quality of Service (QOS)|QOS]]&amp;#039;&amp;#039;&amp;#039; used by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;less resources&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Number of CPUs&amp;#039;&amp;#039;&amp;#039; requested by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;a lower number of CPUs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job Age&amp;#039;&amp;#039;&amp;#039;, i.e. the time that the job has been waiting in the queue&lt;br /&gt;
::: Used to provide higher priority to jobs which have been &amp;#039;&amp;#039;&amp;#039;queued for a longer time&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;FairShare&amp;#039;&amp;#039;&amp;#039;, i.e. a factor computed by SLURM to balance use of the system by different users&lt;br /&gt;
::: Used to provide higher priority to jobs by users who &amp;#039;&amp;#039;&amp;#039;used Mufasa&amp;#039;s resources less than others&amp;#039;&amp;#039;&amp;#039; (see [[#How FairShare works|below]])&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
You can understand how each of these elements contribute to the priority of the jobs waiting for execution (both yours and of other users) with command &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;. A tailored version of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; that only displays the parameters listed above is the following:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sprio -o &amp;quot;%6i %9u %.5Y %.10B %.5Q %.9T %.5A %.5F&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Important note!&lt;br /&gt;
:: The output of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; is only relevant for jobs &amp;#039;&amp;#039;&amp;#039;not yet in execution&amp;#039;&amp;#039;&amp;#039;. For an already running job, the value shown by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; is the priority that the job would get if still queued, which usually does not correspond to its actual priority.&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Here is an example of its output:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
JOBID  USER      PRIOR      ASSOC   QOS      TRES   AGE FAIRS&lt;br /&gt;
5774   itombini   3492       2500   750  cpu=-800   906   136&lt;br /&gt;
5780   fbottazzi  2266        625   750  cpu=-600   420  1071&lt;br /&gt;
5783   mzoccola   1766        625   750  cpu=-800   217   974&lt;br /&gt;
5790   amoglia    2717       2500   375  cpu=-800   156   487&lt;br /&gt;
5791   qliu       2232       2500   375  cpu=-800   138    19&lt;br /&gt;
5792   fpastori   2197       2500   375  cpu=-800    84    39&lt;br /&gt;
5793   bagostini  2034        625  1500 cpu=-1200    18  1091&lt;br /&gt;
5794   lpassoni   2517       2500   750  cpu=-800     9    58&lt;br /&gt;
5795   lpassoni   2517       2500   750  cpu=-800     9    58&lt;br /&gt;
5796   adurat     3101       2500   750  cpu=-800     8   643&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
In it, column ASSOC (i.e., association) corresponds to user category, while column TRES corresponds to the number of CPUs. Note that the contribution of CPUs to priority is &amp;#039;&amp;#039;negative&amp;#039;&amp;#039;, i.e. it is a penalty. Column PRIOR provides the overall priority of each job.&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
=== How FairShare works ===&lt;br /&gt;
In Mufasa 2.0, the FairShare of each user is computed according to these rules:&lt;br /&gt;
# It considers the quantity used by (current and past) user jobs of:&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;CPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of CPUs used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;RAM&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the amount of RAM used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;GPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of GPUs used&lt;br /&gt;
#** note: 40 GB GPUs have double the impact on FairShare than 20 GB GPUs&lt;br /&gt;
# The impact on FairShare of any use of resources is &amp;#039;&amp;#039;&amp;#039;proportional to its duration&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
#* example: using 32 GB of RAM for 48h has the same impact of using 64 GB of RAM for 24h&lt;br /&gt;
# FairShare has a &amp;quot;fading memory&amp;quot;: i.e., resource use has less and less impact on FairShare the farther in the past it is&lt;br /&gt;
#* consequence: over time, the &amp;quot;history&amp;quot; of a user gets forgotten by FairShare&lt;br /&gt;
&lt;br /&gt;
== How to maximise the priority of your jobs ==&lt;br /&gt;
&lt;br /&gt;
Every time you run a SLURM job, follow these guidelines:&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Choose the less powerful QOS compatible with the needs of your job&lt;br /&gt;
:: QOSes with access to less resources have higher base priority&lt;br /&gt;
&lt;br /&gt;
; Only request CPUs and RAM that your job really needs&lt;br /&gt;
:: Usually, code exploits multiple CPUs only if specifically designed to do so: if unsure, only ask for 1 CPU&lt;br /&gt;
:: The fewer resources your job asks for, the less it will wait before they become available&lt;br /&gt;
:: Asking for fewer CPUs and/or less RAM improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Do not request more time than your jobs needs to complete&lt;br /&gt;
:: Make a worst-case estimate and only ask for that duration&lt;br /&gt;
:: Asking for less of Mufasa&amp;#039;s time improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Debug and test code using less powerful QOSes before running it with more powerful QOSes&lt;br /&gt;
:: Your test jobs will get a higher priority and your FairShare will improve&lt;br /&gt;
&lt;br /&gt;
; Cancel jobs that you don&amp;#039;t need them anymore&lt;br /&gt;
:: Always use [[User_Jobs#Cancelling_a_job_with_scancel|scancel]] to delete completed (or crashed) jobs: your Fairshare will improve&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Protip: it&amp;#039;s a good idea to [[User Jobs#Looking for unused GPUs|check for unused GPUs]] before choosing what to request. Requesting a GPU that is currently idle will help your job get executed sooner.&lt;br /&gt;
&lt;br /&gt;
= SLURM partitions =&lt;br /&gt;
&lt;br /&gt;
Partitions are another mechanism provided by SLURM to create different levels of access to system resources. Since in Mufasa 2.0 access to resources is controlled via [[#SLURM Quality of Service (QOS)|QOSes]], partitions are not very relevant. (This is one of the main differences between Mufasa 1.0 and Mufasa 2.0.)&lt;br /&gt;
&lt;br /&gt;
Note, however, that the default values for some features of SLURM jobs (e.g., duration) are [[#Default values|set by the partition]].&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, there is a single SLURM partition, called &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt;, and all jobs run on it. The state of &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt; can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sinfo -o &amp;quot;%10P %5a %9T %11L %10l&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PARTITION  AVAIL STATE     DEFAULTTIME TIMELIMIT &lt;br /&gt;
jobs*      up    idle      1:00:00     3-00:00:00&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where columns correspond to the following information:&lt;br /&gt;
&lt;br /&gt;
:; PARTITION&lt;br /&gt;
:: name of the partition; the asterisks indicates that it&amp;#039;s the default one&lt;br /&gt;
&lt;br /&gt;
:; AVAIL&lt;br /&gt;
:: state/availability of the partition: see [[#Partition availability|below]]&lt;br /&gt;
&lt;br /&gt;
:; STATE&lt;br /&gt;
:: state (using [https://slurm.schedmd.com/sinfo.html#SECTION_NODE-STATE-CODES these codes])&lt;br /&gt;
:: typical values are &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;mixed&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that some of the resources are busy executing jobs while other are idle, and &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;allocated&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that all of the resources are in use&lt;br /&gt;
&lt;br /&gt;
:; DEFAULTTIME&lt;br /&gt;
:: default runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:; TIMELIMIT&lt;br /&gt;
:: maximum runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
The asterisk at the end of the partition name indicates the default partition, i.e. the one on which jobs which do not ask for a specific partition are run.&lt;br /&gt;
&lt;br /&gt;
Command &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; does not tell you about the &amp;#039;&amp;#039;jobs&amp;#039;&amp;#039; submitted to a partition. This information is obtained, instead, with [[User Jobs#Inspecting jobs with squeue|command &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;]].&lt;br /&gt;
&lt;br /&gt;
== Partition availability ==&lt;br /&gt;
&lt;br /&gt;
The most important information that &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; provides is the &amp;#039;&amp;#039;&amp;#039;availability&amp;#039;&amp;#039;&amp;#039; (also called &amp;#039;&amp;#039;&amp;#039;state&amp;#039;&amp;#039;&amp;#039;) of partitions. This is shown in column &amp;quot;AVAIL&amp;quot;. Possible partition states are:&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is available&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed as soon as resources allow&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is in the process of becoming unavailable (i.e., of entering the &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; state: see below)&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is unavailable&lt;br /&gt;
:: There are no running jobs&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
When a partition goes from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; no harm is done to running jobs. In a normally functioning SLURM system, the passage from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; happens only when no jobs are running on the partition. If (e.g., due to a malfunction) the passage happens with jobs still running, they get killed.&lt;br /&gt;
&lt;br /&gt;
A partition in state &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; requires intervention by a [[Roles|Job Administrator]] to be restored to &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
== Default values ==&lt;br /&gt;
&lt;br /&gt;
The features of SLURM partitions, including the &amp;#039;&amp;#039;&amp;#039;default values&amp;#039;&amp;#039;&amp;#039; which are applied to jobs that do not make explicit requests, can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scontrol show partition&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PartitionName=jobs&lt;br /&gt;
   AllowGroups=ALL AllowAccounts=ALL AllowQos=nogpu,gpulight,gpu,gpuwide,gpuheavy-20,gpuheavy-40&lt;br /&gt;
   AllocNodes=ALL Default=YES QoS=N/A&lt;br /&gt;
=&amp;gt; DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO&lt;br /&gt;
   MaxNodes=UNLIMITED MaxTime=3-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED&lt;br /&gt;
   Nodes=gn01&lt;br /&gt;
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO&lt;br /&gt;
   OverTimeLimit=NONE PreemptMode=OFF&lt;br /&gt;
   State=UP TotalCPUs=48 TotalNodes=1 SelectTypeParameters=NONE&lt;br /&gt;
   JobDefaults=(null)&lt;br /&gt;
=&amp;gt; DefMemPerNode=4096 MaxMemPerNode=UNLIMITED&lt;br /&gt;
   TRES=cpu=48,mem=1011435M,node=1,billing=49,gres/gpu=13,gres/gpu:3g.20gb=5,gres/gpu:40gb=3,gres/gpu:4g.20gb=5&lt;br /&gt;
   TRESBillingWeights=cpu=1.0,gres/gpu:3g.20gb=6.0,gres/gpu:4g.20gb=6.0,gres/gpu:40gb=6.0,mem=0.05g&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In the example, we have highlighted with  &amp;#039;&amp;#039;&amp;#039;=&amp;gt;&amp;#039;&amp;#039;&amp;#039;  the most relevant default values for Mufasa users, i.e.:&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefaultTime&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default execution time assigned to a job run on the partition (e.g., 1 hour)&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefMemPerNode&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default amount of RAM assigned to a job run on the partition (e.g., 4GB)&lt;br /&gt;
&lt;br /&gt;
= System resources subjected to limitations =&lt;br /&gt;
&lt;br /&gt;
In systems based on SLURM like Mufasa, &amp;#039;&amp;#039;&amp;#039;TRES (Trackable RESources)&amp;#039;&amp;#039;&amp;#039; are (from [https://slurm.schedmd.com/tres.html SLURM&amp;#039;s documentation] &amp;quot;&amp;#039;&amp;#039;resources that can be tracked for usage or used to enforce limits against.&amp;#039;&amp;#039;&amp;quot;&lt;br /&gt;
&lt;br /&gt;
TRES include CPUs, RAM and &amp;#039;&amp;#039;&amp;#039;GRES&amp;#039;&amp;#039;&amp;#039;. The last term stands for &amp;#039;&amp;#039;Generic RESources&amp;#039;&amp;#039; that a job may need for its execution. In Mufasa, the only &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; resources are the GPUs.&lt;br /&gt;
&lt;br /&gt;
== &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax ==&lt;br /&gt;
&lt;br /&gt;
To ask SLURM to assign GRES resources (i.e., GPUs) to a job, a special syntax must be used. Precisely, the name of each GPU resource takes the form&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:Name:Type&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
Considering the [[System#CPUs and GPUs|GPU complement of Mufasa]], &amp;lt;code&amp;gt;Type&amp;lt;/code&amp;gt; takes the following values:&lt;br /&gt;
&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:40gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 40 Gbytes of RAM&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:4g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 4 compute units&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 3 compute units&lt;br /&gt;
&lt;br /&gt;
So, for instance,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
identifies a resource corresponding to a GPU with 20 GB of RAM and 3 compute units.&lt;br /&gt;
&lt;br /&gt;
When asking for a GRES resource (e.g., in an &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command or an &amp;lt;code&amp;gt;SBATCH&amp;lt;/code&amp;gt; directive of an [[User Jobs#Using execution scripts to run jobs|execution script]]), the syntax required by SLURM is&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:&amp;lt;Type&amp;gt;:&amp;lt;Quantity&amp;gt;&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
where &amp;lt;code&amp;gt;Quantity&amp;lt;/code&amp;gt; is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of type &amp;lt;code&amp;gt;4g.20gb&amp;lt;/code&amp;gt; the syntax is&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:4g.20gb:2&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SLURM&amp;#039;s &amp;#039;&amp;#039;generic resources&amp;#039;&amp;#039; are defined in &amp;lt;code&amp;gt;/etc/slurm/gres.conf&amp;lt;/code&amp;gt;. In order to make GPUs available to SLURM&amp;#039;s &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; management, Mufasa makes use of Nvidia&amp;#039;s [https://developer.nvidia.com/nvidia-management-library-nvml NVML library]. For additional information see [https://slurm.schedmd.com/gres.html SLURM&amp;#039;s documentation].&lt;/div&gt;</summary>
		<author><name>GiulioFontana</name></author>
	</entry>
	<entry>
		<id>https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2575</id>
		<title>SLURM</title>
		<link rel="alternate" type="text/html" href="https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2575"/>
		<updated>2026-11-03T14:40:38Z</updated>

		<summary type="html">&lt;p&gt;GiulioFontana: /* Elements determining job priority */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page presents the features of SLURM that are most relevant to Mufasa&amp;#039;s [[Roles|Job Users]]. Job Users can submit jobs for execution, cancel their own jobs, and see other users&amp;#039; jobs (but not intervene on them).&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa &amp;#039;&amp;#039;&amp;#039;must use SLURM&amp;#039;&amp;#039;&amp;#039; to run resource-heavy processes, i.e. computing jobs that require one or more of the following:&lt;br /&gt;
* GPUs&lt;br /&gt;
* multiple CPUs&lt;br /&gt;
* powerful CPUs&lt;br /&gt;
* a significant amount of RAM&lt;br /&gt;
&lt;br /&gt;
In fact, only processes run via SLURM have access to all the resources of Mufasa. Processes run outside SLURM are executed by the [[System#Login server|login server]] virtual machine, which has minimal resources and no GPUs. Using SLURM is therefore the only way to execute resource-heavy jobs on Mufasa (this is a key difference between Mufasa 1.0 and Mufasa 2.0).&lt;br /&gt;
&lt;br /&gt;
= SLURM in a nutshell =&lt;br /&gt;
&lt;br /&gt;
Computation jobs on Mufasa needs to be launched via [[System#The SLURM job scheduling system|SLURM]]. SLURM provides jobs with access to the [[#System resources subjected to limitations|physical resources]] of Mufasa, such as CPUs, GPUs and RAM. Thanks to SLURM, processing jobs share system resources, optimising their occupation and availability. &lt;br /&gt;
&lt;br /&gt;
When a user runs a job, the job does not get executed immediately and is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039;. SLURM executes jobs according to their order in the queue: the top job in the queue gets executed as soon as the necessary resources are available, while jobs lower in the queue wait longer. The position of a job in the queue is due to the &amp;#039;&amp;#039;&amp;#039;[[#Job priority|priority]]&amp;#039;&amp;#039;&amp;#039; assigned to it by SLURM, with higher-priority jobs closer to the top. As a general rule:&lt;br /&gt;
&lt;br /&gt;
;: &amp;#039;&amp;#039;&amp;#039;the greater the fraction of Mufasa&amp;#039;s overall resources that a job asks for, the lower the job&amp;#039;s priority will be&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
The priority mechanism is used to encourage users to use Mufasa&amp;#039;s resources (i.e.: GPUs, CPUs, RAM, execution time) in an effective and equitable manner. This page includes a [[#How_to_maximise_the_priority_of_your_jobs|chart explaining how to maximise the priority of your jobs]].&lt;br /&gt;
&lt;br /&gt;
The &amp;#039;&amp;#039;&amp;#039;time&amp;#039;&amp;#039;&amp;#039; available to a job for its execution is controlled by SLURM. When a user requests execution of a job, they must specify the duration of the time slot that the job needs. The job must complete its execution before the end of the requested time slot, because at the end of such slot it gets killed by SLURM.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0 access to system resources is managed via SLURM&amp;#039;s [[#SLURM Quality of Service (QOS)|Quality of Service (QOS)]] mechanism (Mufasa 1.0 used [[#SLURM_partitions|partitions]] instead). While [[User Jobs#Running jobs with SLURM|launching a processing job via SLURM]], the user must always specify the chosen QOS. QOSes differ in the set of resources that they provide access to because each of them is designed to fit a given type of job.&lt;br /&gt;
&lt;br /&gt;
Mufasa sets limits to the number of jobs by the same user. This page includes a [[#Limits on jobs by the same user|table summarising such limits]].&lt;br /&gt;
&lt;br /&gt;
= SLURM Quality of Service (QOS) =&lt;br /&gt;
&lt;br /&gt;
Through &amp;#039;&amp;#039;&amp;#039;Quality of Services&amp;#039;&amp;#039;&amp;#039; (&amp;#039;&amp;#039;&amp;#039;QOSes&amp;#039;&amp;#039;&amp;#039;), SLURM lets system configurators assign a name to a set of related constraints.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, QOSes are used to define different levels of access to the server&amp;#039;s resources. When [[User Jobs|executing a job with SLURM]], a user must always &amp;#039;&amp;#039;&amp;#039;specify the QOS&amp;#039;&amp;#039;&amp;#039; that their job will use: this choice, in turn, determines what resources the job is able to access and influences the [[#Job priority|priority]] of the job.&lt;br /&gt;
&lt;br /&gt;
Mufasa&amp;#039;s QOSes and their features can be inspected with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list qos format=name%-11,priority,MaxSubmit,maxwall,maxtres%-80,mintres%-18&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following (see below for the same data in table form):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
Name          Priority MaxSubmit     MaxWall MaxTRES                                                                          MinTRES            &lt;br /&gt;
----------- ---------- --------- ----------- -------------------------------------------------------------------------------- ------------------ &lt;br /&gt;
normal               0                                                                                                                           &lt;br /&gt;
nogpu                4         1  3-00:00:00 cpu=16,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=128G                    &lt;br /&gt;
gpuheavy-20          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=2,mem=128G             gres/gpu:4g.20gb=1 &lt;br /&gt;
gpuheavy-40          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=1,gres/gpu:4g.20gb=0,mem=128G             gres/gpu:40gb=1    &lt;br /&gt;
gpulight             8         1    12:00:00 cpu=2,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpu                  2         1  1-00:00:00 cpu=8,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpuwide              2         2  1-00:00:00 cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=1,mem=64G              gres/gpu:4g.20gb=1 &lt;br /&gt;
build               32         1    02:00:00 cpu=2,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=16G&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The columns of this output are the following:&lt;br /&gt;
&lt;br /&gt;
:; Name&lt;br /&gt;
:: name of the QOS&lt;br /&gt;
&lt;br /&gt;
:; Priority&lt;br /&gt;
:: priority tier associated to the QOS (higher value = higher priority): see [[#Job priority|Job priority]] for details&lt;br /&gt;
&lt;br /&gt;
:; MaxSubmit&lt;br /&gt;
:: maximum number of jobs from a single user that can be submitted to SLURM with this QOS; submitted jobs include both running and queued jobs&lt;br /&gt;
:: See [[#Limits on jobs by the same user|Limits on jobs by the same user]] for an overview of the limits on jobs set by Mufasa.&lt;br /&gt;
&lt;br /&gt;
:; MaxWall&lt;br /&gt;
:: maximum wall clock duration of the jobs using the QOS (after which they are killed by SLURM), in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
:: For some QOSes these are not set: it means that they are determined by the [[#SLURM partitions|partition]]. Partitions also define the [[#Default values|default duration]] of jobs.&lt;br /&gt;
&lt;br /&gt;
:; MaxTRES&lt;br /&gt;
:: amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) available to a job using the QOS, where&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;cpu=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of CPUs (i.e., processor cores) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of CPUs specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;gres/&amp;#039;&amp;#039;gpu:Type&amp;#039;&amp;#039;=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of GPUs of class &amp;lt;code&amp;gt;&amp;#039;&amp;#039;Type&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; (see [[User Jobs#gres syntax|&amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax]]) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; (for QOSes that allow access to GPUs) if not specified, the job cannot be launched&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;mem=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;G&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum amount of system RAM is &amp;#039;&amp;#039;K&amp;#039;&amp;#039; GBytes&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of RAM specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
&lt;br /&gt;
:; MinTRES&lt;br /&gt;
:: minimum amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) that a job using the QOS must request in order to actually get executed by SLURM.&lt;br /&gt;
:: (If your job does not actually need these resources, you&amp;#039;ve chosen the wrong QOS: you can use one with a higher priority.)&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; QOS is the default one, and exists only to ensure that users always specify a QOS when running a job. Since &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; has zero priority and no resources, a job run using this QOS would never be run.&lt;br /&gt;
&lt;br /&gt;
The information provided by the &amp;lt;code&amp;gt;sacctmgr list qos&amp;lt;/code&amp;gt; command above is summarised by the following table:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! QOS&lt;br /&gt;
! Priority&amp;lt;/br&amp;gt;tier&lt;br /&gt;
! Max&amp;lt;/br&amp;gt;Submit&lt;br /&gt;
! MaxWall&amp;lt;/br&amp;gt;[h]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;CPUs&lt;br /&gt;
! max RAM&amp;lt;/br&amp;gt;[GB]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;3g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;4g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;40GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | build&lt;br /&gt;
| 32&lt;br /&gt;
| 1&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 16&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpulight&lt;br /&gt;
| 8&lt;br /&gt;
| 1&lt;br /&gt;
| 12&lt;br /&gt;
| 2&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | nogpu&lt;br /&gt;
| 4&lt;br /&gt;
| 1&lt;br /&gt;
| 72&lt;br /&gt;
| 16&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpu&lt;br /&gt;
| 2&lt;br /&gt;
| 1&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuwide&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-20&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| 2 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-40&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
A key piece of information in the table above is the &amp;#039;&amp;#039;&amp;#039;priority tier&amp;#039;&amp;#039;&amp;#039; associated to each QOS. In Mufasa 2.0, priority tiers are used to encourage users to use the &amp;#039;&amp;#039;&amp;#039;least powerful QOS that is compatible with their needs&amp;#039;&amp;#039;&amp;#039;, where &amp;quot;powerful&amp;quot; means &amp;quot;rich with resources&amp;quot;. Less powerful QOSes increase the priority of the jobs that use them, so these jobs tend to be executed sooner. See [[#Job priority|Job priority]] for details about how priority affects the execution order of jobs in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
== The &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS ==&lt;br /&gt;
&lt;br /&gt;
This QOS is specifically designed to be used by Mufasa users to &amp;#039;&amp;#039;&amp;#039;quickly build [[System#Containers|container images]]&amp;#039;&amp;#039;&amp;#039;. Its associated priority tier is very high: this, combined with the fact that jobs using this QOS require few resources, means that such jobs usually get executed very soon.&lt;br /&gt;
&lt;br /&gt;
On the other hand, the limited resources and the lack of access to the GPUs make the &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS unsuitable for other tasks.&lt;br /&gt;
&lt;br /&gt;
See [[Singularity#Building Singularity images|Building Singularity images]] for directions about building Singularity container images.&lt;br /&gt;
&lt;br /&gt;
== Restricted QOSes ==&lt;br /&gt;
&lt;br /&gt;
In Mufasa, the most powerful QOSes are reserved to researchers (called &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users: these include academic personnel and Ph.D. students). M.Sc. students, called &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users, cannot use them while running jobs.&lt;br /&gt;
&lt;br /&gt;
See [[#research users and students users|below]] to understand how &amp;#039;&amp;#039;user categories&amp;#039;&amp;#039; work in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
= &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users =&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa belong to two &amp;#039;&amp;#039;&amp;#039;user categories&amp;#039;&amp;#039;&amp;#039;. Categories provide users with different access to  to Mufasa&amp;#039;s resources: the idea being to provide researchers with more access while still letting students benefit from the server&amp;#039;s capabilities.&lt;br /&gt;
&lt;br /&gt;
User categories in Mufasa are the following:&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. academic personnel and Ph.D. students&lt;br /&gt;
::: * can use all QOSes, including the the [[#Restricted QOSes|restricted ones]]&lt;br /&gt;
::: * their jobs have a higher &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a higher number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. M.Sc. students&lt;br /&gt;
::: * cannot use the [[#Restricted QOSes|restricted QOSes]]&lt;br /&gt;
::: * their jobs have a lower &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a lower number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
You can inspect the differences in priority and running jobs between &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association format=&amp;quot;account,priority,maxjobs&amp;quot; | grep -E &amp;#039;Account|research|students&amp;#039;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
   Account   Priority MaxJobs &lt;br /&gt;
  research          4       2 &lt;br /&gt;
  students          1       1&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
To know what limits apply to your own user, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association where user=$USER format=&amp;quot;user,priority,maxjobs,qos%-60&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
      User   Priority MaxJobs QOS                                                          &lt;br /&gt;
---------- ---------- ------- ------------------------------------------------------------ &lt;br /&gt;
    preali          4       2 build,gpu,gpuheavy-20,gpuheavy-40,gpulight,gpuwide,nogpu&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The list under &amp;quot;QOS&amp;quot; shows what QOSes your user is allowed to use when [[User Jobs|running jobs]]. &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users can use all of them, while &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users can only access a subset of them.&lt;br /&gt;
&lt;br /&gt;
= Limits on jobs by the same user =&lt;br /&gt;
&lt;br /&gt;
Mufasa enforces limits on the number of jobs from a single user. Such limits aim at preventing users from &amp;quot;hogging&amp;quot; system resources, and apply to:&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;submitted jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that the user asked SLURM to execute, each of which may currently be either running or queued&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;running jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that are currently in execution&lt;br /&gt;
&lt;br /&gt;
Limits on submitted jobs are set via [[#SLURM Quality of Service (QOS)|QOSes]], while limits on running jobs are set via [[#research users and students users|user category]].&lt;br /&gt;
&lt;br /&gt;
The following table summarises what limits exist:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! &lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;running&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;submitted&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | global limits&amp;lt;br/&amp;gt;(system-wide)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to limits set by individual QOSes (below)&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | limits for&amp;lt;br/&amp;gt;individual&amp;lt;/br&amp;gt;QOSes&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to global limits (above)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for the&amp;#039;&amp;#039; &amp;lt;code&amp;gt;gpuwide&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;QOS&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for each of the other QOSes&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
= Job priority =&lt;br /&gt;
&lt;br /&gt;
Once the execution of a job has been requested, the job is not run immediately: it is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039; by SLURM, together with all the other jobs awaiting execution. The job on top of the queue at any time is the first to be put into execution as soon as the resources it requires are available. &lt;br /&gt;
&lt;br /&gt;
The order of the items in the job queue depends on their &amp;#039;&amp;#039;&amp;#039;priority&amp;#039;&amp;#039;&amp;#039;. The job with the highest priority is put by SLURM on top of the queue, while all other jobs are put in the queue in order of descending priority.&lt;br /&gt;
&lt;br /&gt;
The goal of SLURM is to maximise resource availability: i.e., to ensure the shorter possible wait time before job execution. To achieve this goal, in Mufasa SLURM is configured to &amp;#039;&amp;#039;&amp;#039;encourage users not to ask for resources or execution time that their job doesn&amp;#039;t need&amp;#039;&amp;#039;&amp;#039;. This is done via the priority mechanism: the more resources and/or time a job requests, the lower its priority will be; and the later it will be executed.&lt;br /&gt;
&lt;br /&gt;
Priority management in Mufasa is designed to set up a &amp;#039;&amp;#039;&amp;#039;virtuous cycle&amp;#039;&amp;#039;&amp;#039; where users, by carefully choosing what to ask for, obtain two results:&lt;br /&gt;
* they ensure that their job is executed as soon as possible;&lt;br /&gt;
* they leave as much as possible of Mufasa&amp;#039;s resources free for other users&amp;#039;s jobs.&lt;br /&gt;
&lt;br /&gt;
== Elements determining job priority ==&lt;br /&gt;
In Mufasa, the priority of a job is computed by SLURM according to the following elements:&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#research users and students users|User category]]&amp;#039;&amp;#039;&amp;#039; (i.e., &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;)&lt;br /&gt;
::: Used to provide higher priority to jobs run by &amp;#039;&amp;#039;&amp;#039;research personnel&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#SLURM Quality of Service (QOS)|QOS]]&amp;#039;&amp;#039;&amp;#039; used by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;less resources&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Number of CPUs&amp;#039;&amp;#039;&amp;#039; requested by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;a lower number of CPUs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job Age&amp;#039;&amp;#039;&amp;#039;, i.e. the time that the job has been waiting in the queue&lt;br /&gt;
::: Used to provide higher priority to jobs which have been &amp;#039;&amp;#039;&amp;#039;queued for a longer time&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;FairShare&amp;#039;&amp;#039;&amp;#039;, i.e. a factor computed by SLURM to balance use of the system by different users&lt;br /&gt;
::: Used to provide higher priority to jobs by users who &amp;#039;&amp;#039;&amp;#039;used Mufasa&amp;#039;s resources less than others&amp;#039;&amp;#039;&amp;#039; (see [[#How FairShare works|below]])&lt;br /&gt;
&lt;br /&gt;
You can understand how each of these elements contribute to the priority of the jobs waiting for execution (both yours and of other users) with command &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;. A tailored version of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; that only displays the parameters listed above is the following:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sprio -o &amp;quot;%6i %9u %.5Y %.10B %.5Q %.9T %.5A %.5F&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Important note!&lt;br /&gt;
:: The output of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; is only relevant for jobs &amp;#039;&amp;#039;&amp;#039;not yet in execution&amp;#039;&amp;#039;&amp;#039;. For an already running job, the value shown by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; is the priority that the job would get if still queued, which usually does not correspond to its actual priority.&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Here is an example of its output:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
JOBID  USER      PRIOR      ASSOC   QOS      TRES   AGE FAIRS&lt;br /&gt;
5774   itombini   3492       2500   750  cpu=-800   906   136&lt;br /&gt;
5780   fbottazzi  2266        625   750  cpu=-600   420  1071&lt;br /&gt;
5783   mzoccola   1766        625   750  cpu=-800   217   974&lt;br /&gt;
5790   amoglia    2717       2500   375  cpu=-800   156   487&lt;br /&gt;
5791   qliu       2232       2500   375  cpu=-800   138    19&lt;br /&gt;
5792   fpastori   2197       2500   375  cpu=-800    84    39&lt;br /&gt;
5793   bagostini  2034        625  1500 cpu=-1200    18  1091&lt;br /&gt;
5794   lpassoni   2517       2500   750  cpu=-800     9    58&lt;br /&gt;
5795   lpassoni   2517       2500   750  cpu=-800     9    58&lt;br /&gt;
5796   adurat     3101       2500   750  cpu=-800     8   643&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
In it, column ASSOC (i.e., association) corresponds to user category, while column TRES corresponds to the number of CPUs. Note that the contribution of CPUs to priority is &amp;#039;&amp;#039;negative&amp;#039;&amp;#039;, i.e. it is a penalty. Column PRIOR provides the overall priority of each job.&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
=== How FairShare works ===&lt;br /&gt;
In Mufasa 2.0, the FairShare of each user is computed according to these rules:&lt;br /&gt;
# It considers the quantity used by (current and past) user jobs of:&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;CPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of CPUs used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;RAM&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the amount of RAM used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;GPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of GPUs used&lt;br /&gt;
#** note: 40 GB GPUs have double the impact on FairShare than 20 GB GPUs&lt;br /&gt;
# The impact on FairShare of any use of resources is &amp;#039;&amp;#039;&amp;#039;proportional to its duration&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
#* example: using 32 GB of RAM for 48h has the same impact of using 64 GB of RAM for 24h&lt;br /&gt;
# FairShare has a &amp;quot;fading memory&amp;quot;: i.e., resource use has less and less impact on FairShare the farther in the past it is&lt;br /&gt;
#* consequence: over time, the &amp;quot;history&amp;quot; of a user gets forgotten by FairShare&lt;br /&gt;
&lt;br /&gt;
== How to maximise the priority of your jobs ==&lt;br /&gt;
&lt;br /&gt;
Every time you run a SLURM job, follow these guidelines:&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Choose the less powerful QOS compatible with the needs of your job&lt;br /&gt;
:: QOSes with access to less resources have higher base priority&lt;br /&gt;
&lt;br /&gt;
; Only request CPUs and RAM that your job really needs&lt;br /&gt;
:: Usually, code exploits multiple CPUs only if specifically designed to do so: if unsure, only ask for 1 CPU&lt;br /&gt;
:: The fewer resources your job asks for, the less it will wait before they become available&lt;br /&gt;
:: Asking for fewer CPUs and/or less RAM improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Do not request more time than your jobs needs to complete&lt;br /&gt;
:: Make a worst-case estimate and only ask for that duration&lt;br /&gt;
:: Asking for less of Mufasa&amp;#039;s time improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Debug and test code using less powerful QOSes before running it with more powerful QOSes&lt;br /&gt;
:: Your test jobs will get a higher priority and your FairShare will improve&lt;br /&gt;
&lt;br /&gt;
; Cancel jobs that you don&amp;#039;t need them anymore&lt;br /&gt;
:: Always use [[User_Jobs#Cancelling_a_job_with_scancel|scancel]] to delete completed (or crashed) jobs: your Fairshare will improve&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Protip: it&amp;#039;s a good idea to [[User Jobs#Looking for unused GPUs|check for unused GPUs]] before choosing what to request. Requesting a GPU that is currently idle will help your job get executed sooner.&lt;br /&gt;
&lt;br /&gt;
= SLURM partitions =&lt;br /&gt;
&lt;br /&gt;
Partitions are another mechanism provided by SLURM to create different levels of access to system resources. Since in Mufasa 2.0 access to resources is controlled via [[#SLURM Quality of Service (QOS)|QOSes]], partitions are not very relevant. (This is one of the main differences between Mufasa 1.0 and Mufasa 2.0.)&lt;br /&gt;
&lt;br /&gt;
Note, however, that the default values for some features of SLURM jobs (e.g., duration) are [[#Default values|set by the partition]].&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, there is a single SLURM partition, called &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt;, and all jobs run on it. The state of &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt; can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sinfo -o &amp;quot;%10P %5a %9T %11L %10l&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PARTITION  AVAIL STATE     DEFAULTTIME TIMELIMIT &lt;br /&gt;
jobs*      up    idle      1:00:00     3-00:00:00&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where columns correspond to the following information:&lt;br /&gt;
&lt;br /&gt;
:; PARTITION&lt;br /&gt;
:: name of the partition; the asterisks indicates that it&amp;#039;s the default one&lt;br /&gt;
&lt;br /&gt;
:; AVAIL&lt;br /&gt;
:: state/availability of the partition: see [[#Partition availability|below]]&lt;br /&gt;
&lt;br /&gt;
:; STATE&lt;br /&gt;
:: state (using [https://slurm.schedmd.com/sinfo.html#SECTION_NODE-STATE-CODES these codes])&lt;br /&gt;
:: typical values are &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;mixed&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that some of the resources are busy executing jobs while other are idle, and &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;allocated&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that all of the resources are in use&lt;br /&gt;
&lt;br /&gt;
:; DEFAULTTIME&lt;br /&gt;
:: default runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:; TIMELIMIT&lt;br /&gt;
:: maximum runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
The asterisk at the end of the partition name indicates the default partition, i.e. the one on which jobs which do not ask for a specific partition are run.&lt;br /&gt;
&lt;br /&gt;
Command &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; does not tell you about the &amp;#039;&amp;#039;jobs&amp;#039;&amp;#039; submitted to a partition. This information is obtained, instead, with [[User Jobs#Inspecting jobs with squeue|command &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;]].&lt;br /&gt;
&lt;br /&gt;
== Partition availability ==&lt;br /&gt;
&lt;br /&gt;
The most important information that &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; provides is the &amp;#039;&amp;#039;&amp;#039;availability&amp;#039;&amp;#039;&amp;#039; (also called &amp;#039;&amp;#039;&amp;#039;state&amp;#039;&amp;#039;&amp;#039;) of partitions. This is shown in column &amp;quot;AVAIL&amp;quot;. Possible partition states are:&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is available&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed as soon as resources allow&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is in the process of becoming unavailable (i.e., of entering the &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; state: see below)&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is unavailable&lt;br /&gt;
:: There are no running jobs&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
When a partition goes from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; no harm is done to running jobs. In a normally functioning SLURM system, the passage from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; happens only when no jobs are running on the partition. If (e.g., due to a malfunction) the passage happens with jobs still running, they get killed.&lt;br /&gt;
&lt;br /&gt;
A partition in state &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; requires intervention by a [[Roles|Job Administrator]] to be restored to &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
== Default values ==&lt;br /&gt;
&lt;br /&gt;
The features of SLURM partitions, including the &amp;#039;&amp;#039;&amp;#039;default values&amp;#039;&amp;#039;&amp;#039; which are applied to jobs that do not make explicit requests, can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scontrol show partition&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PartitionName=jobs&lt;br /&gt;
   AllowGroups=ALL AllowAccounts=ALL AllowQos=nogpu,gpulight,gpu,gpuwide,gpuheavy-20,gpuheavy-40&lt;br /&gt;
   AllocNodes=ALL Default=YES QoS=N/A&lt;br /&gt;
=&amp;gt; DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO&lt;br /&gt;
   MaxNodes=UNLIMITED MaxTime=3-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED&lt;br /&gt;
   Nodes=gn01&lt;br /&gt;
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO&lt;br /&gt;
   OverTimeLimit=NONE PreemptMode=OFF&lt;br /&gt;
   State=UP TotalCPUs=48 TotalNodes=1 SelectTypeParameters=NONE&lt;br /&gt;
   JobDefaults=(null)&lt;br /&gt;
=&amp;gt; DefMemPerNode=4096 MaxMemPerNode=UNLIMITED&lt;br /&gt;
   TRES=cpu=48,mem=1011435M,node=1,billing=49,gres/gpu=13,gres/gpu:3g.20gb=5,gres/gpu:40gb=3,gres/gpu:4g.20gb=5&lt;br /&gt;
   TRESBillingWeights=cpu=1.0,gres/gpu:3g.20gb=6.0,gres/gpu:4g.20gb=6.0,gres/gpu:40gb=6.0,mem=0.05g&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In the example, we have highlighted with  &amp;#039;&amp;#039;&amp;#039;=&amp;gt;&amp;#039;&amp;#039;&amp;#039;  the most relevant default values for Mufasa users, i.e.:&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefaultTime&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default execution time assigned to a job run on the partition (e.g., 1 hour)&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefMemPerNode&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default amount of RAM assigned to a job run on the partition (e.g., 4GB)&lt;br /&gt;
&lt;br /&gt;
= System resources subjected to limitations =&lt;br /&gt;
&lt;br /&gt;
In systems based on SLURM like Mufasa, &amp;#039;&amp;#039;&amp;#039;TRES (Trackable RESources)&amp;#039;&amp;#039;&amp;#039; are (from [https://slurm.schedmd.com/tres.html SLURM&amp;#039;s documentation] &amp;quot;&amp;#039;&amp;#039;resources that can be tracked for usage or used to enforce limits against.&amp;#039;&amp;#039;&amp;quot;&lt;br /&gt;
&lt;br /&gt;
TRES include CPUs, RAM and &amp;#039;&amp;#039;&amp;#039;GRES&amp;#039;&amp;#039;&amp;#039;. The last term stands for &amp;#039;&amp;#039;Generic RESources&amp;#039;&amp;#039; that a job may need for its execution. In Mufasa, the only &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; resources are the GPUs.&lt;br /&gt;
&lt;br /&gt;
== &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax ==&lt;br /&gt;
&lt;br /&gt;
To ask SLURM to assign GRES resources (i.e., GPUs) to a job, a special syntax must be used. Precisely, the name of each GPU resource takes the form&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:Name:Type&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
Considering the [[System#CPUs and GPUs|GPU complement of Mufasa]], &amp;lt;code&amp;gt;Type&amp;lt;/code&amp;gt; takes the following values:&lt;br /&gt;
&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:40gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 40 Gbytes of RAM&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:4g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 4 compute units&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 3 compute units&lt;br /&gt;
&lt;br /&gt;
So, for instance,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
identifies a resource corresponding to a GPU with 20 GB of RAM and 3 compute units.&lt;br /&gt;
&lt;br /&gt;
When asking for a GRES resource (e.g., in an &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command or an &amp;lt;code&amp;gt;SBATCH&amp;lt;/code&amp;gt; directive of an [[User Jobs#Using execution scripts to run jobs|execution script]]), the syntax required by SLURM is&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:&amp;lt;Type&amp;gt;:&amp;lt;Quantity&amp;gt;&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
where &amp;lt;code&amp;gt;Quantity&amp;lt;/code&amp;gt; is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of type &amp;lt;code&amp;gt;4g.20gb&amp;lt;/code&amp;gt; the syntax is&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:4g.20gb:2&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SLURM&amp;#039;s &amp;#039;&amp;#039;generic resources&amp;#039;&amp;#039; are defined in &amp;lt;code&amp;gt;/etc/slurm/gres.conf&amp;lt;/code&amp;gt;. In order to make GPUs available to SLURM&amp;#039;s &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; management, Mufasa makes use of Nvidia&amp;#039;s [https://developer.nvidia.com/nvidia-management-library-nvml NVML library]. For additional information see [https://slurm.schedmd.com/gres.html SLURM&amp;#039;s documentation].&lt;/div&gt;</summary>
		<author><name>GiulioFontana</name></author>
	</entry>
	<entry>
		<id>https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2574</id>
		<title>SLURM</title>
		<link rel="alternate" type="text/html" href="https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2574"/>
		<updated>2026-11-03T14:38:34Z</updated>

		<summary type="html">&lt;p&gt;GiulioFontana: /* Elements determining job priority */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page presents the features of SLURM that are most relevant to Mufasa&amp;#039;s [[Roles|Job Users]]. Job Users can submit jobs for execution, cancel their own jobs, and see other users&amp;#039; jobs (but not intervene on them).&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa &amp;#039;&amp;#039;&amp;#039;must use SLURM&amp;#039;&amp;#039;&amp;#039; to run resource-heavy processes, i.e. computing jobs that require one or more of the following:&lt;br /&gt;
* GPUs&lt;br /&gt;
* multiple CPUs&lt;br /&gt;
* powerful CPUs&lt;br /&gt;
* a significant amount of RAM&lt;br /&gt;
&lt;br /&gt;
In fact, only processes run via SLURM have access to all the resources of Mufasa. Processes run outside SLURM are executed by the [[System#Login server|login server]] virtual machine, which has minimal resources and no GPUs. Using SLURM is therefore the only way to execute resource-heavy jobs on Mufasa (this is a key difference between Mufasa 1.0 and Mufasa 2.0).&lt;br /&gt;
&lt;br /&gt;
= SLURM in a nutshell =&lt;br /&gt;
&lt;br /&gt;
Computation jobs on Mufasa needs to be launched via [[System#The SLURM job scheduling system|SLURM]]. SLURM provides jobs with access to the [[#System resources subjected to limitations|physical resources]] of Mufasa, such as CPUs, GPUs and RAM. Thanks to SLURM, processing jobs share system resources, optimising their occupation and availability. &lt;br /&gt;
&lt;br /&gt;
When a user runs a job, the job does not get executed immediately and is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039;. SLURM executes jobs according to their order in the queue: the top job in the queue gets executed as soon as the necessary resources are available, while jobs lower in the queue wait longer. The position of a job in the queue is due to the &amp;#039;&amp;#039;&amp;#039;[[#Job priority|priority]]&amp;#039;&amp;#039;&amp;#039; assigned to it by SLURM, with higher-priority jobs closer to the top. As a general rule:&lt;br /&gt;
&lt;br /&gt;
;: &amp;#039;&amp;#039;&amp;#039;the greater the fraction of Mufasa&amp;#039;s overall resources that a job asks for, the lower the job&amp;#039;s priority will be&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
The priority mechanism is used to encourage users to use Mufasa&amp;#039;s resources (i.e.: GPUs, CPUs, RAM, execution time) in an effective and equitable manner. This page includes a [[#How_to_maximise_the_priority_of_your_jobs|chart explaining how to maximise the priority of your jobs]].&lt;br /&gt;
&lt;br /&gt;
The &amp;#039;&amp;#039;&amp;#039;time&amp;#039;&amp;#039;&amp;#039; available to a job for its execution is controlled by SLURM. When a user requests execution of a job, they must specify the duration of the time slot that the job needs. The job must complete its execution before the end of the requested time slot, because at the end of such slot it gets killed by SLURM.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0 access to system resources is managed via SLURM&amp;#039;s [[#SLURM Quality of Service (QOS)|Quality of Service (QOS)]] mechanism (Mufasa 1.0 used [[#SLURM_partitions|partitions]] instead). While [[User Jobs#Running jobs with SLURM|launching a processing job via SLURM]], the user must always specify the chosen QOS. QOSes differ in the set of resources that they provide access to because each of them is designed to fit a given type of job.&lt;br /&gt;
&lt;br /&gt;
Mufasa sets limits to the number of jobs by the same user. This page includes a [[#Limits on jobs by the same user|table summarising such limits]].&lt;br /&gt;
&lt;br /&gt;
= SLURM Quality of Service (QOS) =&lt;br /&gt;
&lt;br /&gt;
Through &amp;#039;&amp;#039;&amp;#039;Quality of Services&amp;#039;&amp;#039;&amp;#039; (&amp;#039;&amp;#039;&amp;#039;QOSes&amp;#039;&amp;#039;&amp;#039;), SLURM lets system configurators assign a name to a set of related constraints.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, QOSes are used to define different levels of access to the server&amp;#039;s resources. When [[User Jobs|executing a job with SLURM]], a user must always &amp;#039;&amp;#039;&amp;#039;specify the QOS&amp;#039;&amp;#039;&amp;#039; that their job will use: this choice, in turn, determines what resources the job is able to access and influences the [[#Job priority|priority]] of the job.&lt;br /&gt;
&lt;br /&gt;
Mufasa&amp;#039;s QOSes and their features can be inspected with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list qos format=name%-11,priority,MaxSubmit,maxwall,maxtres%-80,mintres%-18&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following (see below for the same data in table form):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
Name          Priority MaxSubmit     MaxWall MaxTRES                                                                          MinTRES            &lt;br /&gt;
----------- ---------- --------- ----------- -------------------------------------------------------------------------------- ------------------ &lt;br /&gt;
normal               0                                                                                                                           &lt;br /&gt;
nogpu                4         1  3-00:00:00 cpu=16,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=128G                    &lt;br /&gt;
gpuheavy-20          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=2,mem=128G             gres/gpu:4g.20gb=1 &lt;br /&gt;
gpuheavy-40          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=1,gres/gpu:4g.20gb=0,mem=128G             gres/gpu:40gb=1    &lt;br /&gt;
gpulight             8         1    12:00:00 cpu=2,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpu                  2         1  1-00:00:00 cpu=8,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpuwide              2         2  1-00:00:00 cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=1,mem=64G              gres/gpu:4g.20gb=1 &lt;br /&gt;
build               32         1    02:00:00 cpu=2,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=16G&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The columns of this output are the following:&lt;br /&gt;
&lt;br /&gt;
:; Name&lt;br /&gt;
:: name of the QOS&lt;br /&gt;
&lt;br /&gt;
:; Priority&lt;br /&gt;
:: priority tier associated to the QOS (higher value = higher priority): see [[#Job priority|Job priority]] for details&lt;br /&gt;
&lt;br /&gt;
:; MaxSubmit&lt;br /&gt;
:: maximum number of jobs from a single user that can be submitted to SLURM with this QOS; submitted jobs include both running and queued jobs&lt;br /&gt;
:: See [[#Limits on jobs by the same user|Limits on jobs by the same user]] for an overview of the limits on jobs set by Mufasa.&lt;br /&gt;
&lt;br /&gt;
:; MaxWall&lt;br /&gt;
:: maximum wall clock duration of the jobs using the QOS (after which they are killed by SLURM), in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
:: For some QOSes these are not set: it means that they are determined by the [[#SLURM partitions|partition]]. Partitions also define the [[#Default values|default duration]] of jobs.&lt;br /&gt;
&lt;br /&gt;
:; MaxTRES&lt;br /&gt;
:: amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) available to a job using the QOS, where&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;cpu=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of CPUs (i.e., processor cores) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of CPUs specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;gres/&amp;#039;&amp;#039;gpu:Type&amp;#039;&amp;#039;=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of GPUs of class &amp;lt;code&amp;gt;&amp;#039;&amp;#039;Type&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; (see [[User Jobs#gres syntax|&amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax]]) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; (for QOSes that allow access to GPUs) if not specified, the job cannot be launched&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;mem=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;G&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum amount of system RAM is &amp;#039;&amp;#039;K&amp;#039;&amp;#039; GBytes&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of RAM specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
&lt;br /&gt;
:; MinTRES&lt;br /&gt;
:: minimum amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) that a job using the QOS must request in order to actually get executed by SLURM.&lt;br /&gt;
:: (If your job does not actually need these resources, you&amp;#039;ve chosen the wrong QOS: you can use one with a higher priority.)&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; QOS is the default one, and exists only to ensure that users always specify a QOS when running a job. Since &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; has zero priority and no resources, a job run using this QOS would never be run.&lt;br /&gt;
&lt;br /&gt;
The information provided by the &amp;lt;code&amp;gt;sacctmgr list qos&amp;lt;/code&amp;gt; command above is summarised by the following table:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! QOS&lt;br /&gt;
! Priority&amp;lt;/br&amp;gt;tier&lt;br /&gt;
! Max&amp;lt;/br&amp;gt;Submit&lt;br /&gt;
! MaxWall&amp;lt;/br&amp;gt;[h]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;CPUs&lt;br /&gt;
! max RAM&amp;lt;/br&amp;gt;[GB]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;3g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;4g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;40GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | build&lt;br /&gt;
| 32&lt;br /&gt;
| 1&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 16&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpulight&lt;br /&gt;
| 8&lt;br /&gt;
| 1&lt;br /&gt;
| 12&lt;br /&gt;
| 2&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | nogpu&lt;br /&gt;
| 4&lt;br /&gt;
| 1&lt;br /&gt;
| 72&lt;br /&gt;
| 16&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpu&lt;br /&gt;
| 2&lt;br /&gt;
| 1&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuwide&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-20&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| 2 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-40&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
A key piece of information in the table above is the &amp;#039;&amp;#039;&amp;#039;priority tier&amp;#039;&amp;#039;&amp;#039; associated to each QOS. In Mufasa 2.0, priority tiers are used to encourage users to use the &amp;#039;&amp;#039;&amp;#039;least powerful QOS that is compatible with their needs&amp;#039;&amp;#039;&amp;#039;, where &amp;quot;powerful&amp;quot; means &amp;quot;rich with resources&amp;quot;. Less powerful QOSes increase the priority of the jobs that use them, so these jobs tend to be executed sooner. See [[#Job priority|Job priority]] for details about how priority affects the execution order of jobs in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
== The &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS ==&lt;br /&gt;
&lt;br /&gt;
This QOS is specifically designed to be used by Mufasa users to &amp;#039;&amp;#039;&amp;#039;quickly build [[System#Containers|container images]]&amp;#039;&amp;#039;&amp;#039;. Its associated priority tier is very high: this, combined with the fact that jobs using this QOS require few resources, means that such jobs usually get executed very soon.&lt;br /&gt;
&lt;br /&gt;
On the other hand, the limited resources and the lack of access to the GPUs make the &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS unsuitable for other tasks.&lt;br /&gt;
&lt;br /&gt;
See [[Singularity#Building Singularity images|Building Singularity images]] for directions about building Singularity container images.&lt;br /&gt;
&lt;br /&gt;
== Restricted QOSes ==&lt;br /&gt;
&lt;br /&gt;
In Mufasa, the most powerful QOSes are reserved to researchers (called &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users: these include academic personnel and Ph.D. students). M.Sc. students, called &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users, cannot use them while running jobs.&lt;br /&gt;
&lt;br /&gt;
See [[#research users and students users|below]] to understand how &amp;#039;&amp;#039;user categories&amp;#039;&amp;#039; work in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
= &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users =&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa belong to two &amp;#039;&amp;#039;&amp;#039;user categories&amp;#039;&amp;#039;&amp;#039;. Categories provide users with different access to  to Mufasa&amp;#039;s resources: the idea being to provide researchers with more access while still letting students benefit from the server&amp;#039;s capabilities.&lt;br /&gt;
&lt;br /&gt;
User categories in Mufasa are the following:&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. academic personnel and Ph.D. students&lt;br /&gt;
::: * can use all QOSes, including the the [[#Restricted QOSes|restricted ones]]&lt;br /&gt;
::: * their jobs have a higher &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a higher number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. M.Sc. students&lt;br /&gt;
::: * cannot use the [[#Restricted QOSes|restricted QOSes]]&lt;br /&gt;
::: * their jobs have a lower &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a lower number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
You can inspect the differences in priority and running jobs between &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association format=&amp;quot;account,priority,maxjobs&amp;quot; | grep -E &amp;#039;Account|research|students&amp;#039;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
   Account   Priority MaxJobs &lt;br /&gt;
  research          4       2 &lt;br /&gt;
  students          1       1&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
To know what limits apply to your own user, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association where user=$USER format=&amp;quot;user,priority,maxjobs,qos%-60&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
      User   Priority MaxJobs QOS                                                          &lt;br /&gt;
---------- ---------- ------- ------------------------------------------------------------ &lt;br /&gt;
    preali          4       2 build,gpu,gpuheavy-20,gpuheavy-40,gpulight,gpuwide,nogpu&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The list under &amp;quot;QOS&amp;quot; shows what QOSes your user is allowed to use when [[User Jobs|running jobs]]. &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users can use all of them, while &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users can only access a subset of them.&lt;br /&gt;
&lt;br /&gt;
= Limits on jobs by the same user =&lt;br /&gt;
&lt;br /&gt;
Mufasa enforces limits on the number of jobs from a single user. Such limits aim at preventing users from &amp;quot;hogging&amp;quot; system resources, and apply to:&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;submitted jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that the user asked SLURM to execute, each of which may currently be either running or queued&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;running jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that are currently in execution&lt;br /&gt;
&lt;br /&gt;
Limits on submitted jobs are set via [[#SLURM Quality of Service (QOS)|QOSes]], while limits on running jobs are set via [[#research users and students users|user category]].&lt;br /&gt;
&lt;br /&gt;
The following table summarises what limits exist:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! &lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;running&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;submitted&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | global limits&amp;lt;br/&amp;gt;(system-wide)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to limits set by individual QOSes (below)&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | limits for&amp;lt;br/&amp;gt;individual&amp;lt;/br&amp;gt;QOSes&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to global limits (above)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for the&amp;#039;&amp;#039; &amp;lt;code&amp;gt;gpuwide&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;QOS&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for each of the other QOSes&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
= Job priority =&lt;br /&gt;
&lt;br /&gt;
Once the execution of a job has been requested, the job is not run immediately: it is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039; by SLURM, together with all the other jobs awaiting execution. The job on top of the queue at any time is the first to be put into execution as soon as the resources it requires are available. &lt;br /&gt;
&lt;br /&gt;
The order of the items in the job queue depends on their &amp;#039;&amp;#039;&amp;#039;priority&amp;#039;&amp;#039;&amp;#039;. The job with the highest priority is put by SLURM on top of the queue, while all other jobs are put in the queue in order of descending priority.&lt;br /&gt;
&lt;br /&gt;
The goal of SLURM is to maximise resource availability: i.e., to ensure the shorter possible wait time before job execution. To achieve this goal, in Mufasa SLURM is configured to &amp;#039;&amp;#039;&amp;#039;encourage users not to ask for resources or execution time that their job doesn&amp;#039;t need&amp;#039;&amp;#039;&amp;#039;. This is done via the priority mechanism: the more resources and/or time a job requests, the lower its priority will be; and the later it will be executed.&lt;br /&gt;
&lt;br /&gt;
Priority management in Mufasa is designed to set up a &amp;#039;&amp;#039;&amp;#039;virtuous cycle&amp;#039;&amp;#039;&amp;#039; where users, by carefully choosing what to ask for, obtain two results:&lt;br /&gt;
* they ensure that their job is executed as soon as possible;&lt;br /&gt;
* they leave as much as possible of Mufasa&amp;#039;s resources free for other users&amp;#039;s jobs.&lt;br /&gt;
&lt;br /&gt;
== Elements determining job priority ==&lt;br /&gt;
In Mufasa, the priority of a job is computed by SLURM according to the following elements:&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#research users and students users|User category]]&amp;#039;&amp;#039;&amp;#039; (i.e., &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;)&lt;br /&gt;
::: Used to provide higher priority to jobs run by &amp;#039;&amp;#039;&amp;#039;research personnel&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#SLURM Quality of Service (QOS)|QOS]]&amp;#039;&amp;#039;&amp;#039; used by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;less resources&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Number of CPUs&amp;#039;&amp;#039;&amp;#039; requested by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;a lower number of CPUs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job Age&amp;#039;&amp;#039;&amp;#039;, i.e. the time that the job has been waiting in the queue&lt;br /&gt;
::: Used to provide higher priority to jobs which have been &amp;#039;&amp;#039;&amp;#039;queued for a longer time&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;FairShare&amp;#039;&amp;#039;&amp;#039;, i.e. a factor computed by SLURM to balance use of the system by different users&lt;br /&gt;
::: Used to provide higher priority to jobs by users who &amp;#039;&amp;#039;&amp;#039;used Mufasa&amp;#039;s resources less than others&amp;#039;&amp;#039;&amp;#039; (see [[#How FairShare works|below]])&lt;br /&gt;
&lt;br /&gt;
You can understand how each of these elements contribute to the priority of your jobs waiting for execution with command &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;. A tailored version of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; that only displays the parameters listed above is the following:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sprio -o &amp;quot;%6i %9u %.5Y %.10B %.5Q %.9T %.5A %.5F&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Important note!&lt;br /&gt;
:: The output of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; is only relevant for jobs &amp;#039;&amp;#039;&amp;#039;not yet in execution&amp;#039;&amp;#039;&amp;#039;. For an already running job, the value shown by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; is the priority that the job would get if still queued, which usually does not correspond to its actual priority.&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Here is an example of its output:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
JOBID  USER      PRIOR      ASSOC   QOS      TRES   AGE FAIRS&lt;br /&gt;
5774   itombini   3492       2500   750  cpu=-800   906   136&lt;br /&gt;
5780   fbottazzi  2266        625   750  cpu=-600   420  1071&lt;br /&gt;
5783   mzoccola   1766        625   750  cpu=-800   217   974&lt;br /&gt;
5790   amoglia    2717       2500   375  cpu=-800   156   487&lt;br /&gt;
5791   qliu       2232       2500   375  cpu=-800   138    19&lt;br /&gt;
5792   fpastori   2197       2500   375  cpu=-800    84    39&lt;br /&gt;
5793   bagostini  2034        625  1500 cpu=-1200    18  1091&lt;br /&gt;
5794   lpassoni   2517       2500   750  cpu=-800     9    58&lt;br /&gt;
5795   lpassoni   2517       2500   750  cpu=-800     9    58&lt;br /&gt;
5796   adurat     3101       2500   750  cpu=-800     8   643&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
In it, column ASSOC (i.e., association) corresponds to user category, while column TRES corresponds to the number of CPUs. Note that the contribution of CPUs to priority is &amp;#039;&amp;#039;negative&amp;#039;&amp;#039;, i.e. it is a penalty. Column PRIOR provides the overall priority of each job.&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
=== How FairShare works ===&lt;br /&gt;
In Mufasa 2.0, the FairShare of each user is computed according to these rules:&lt;br /&gt;
# It considers the quantity used by (current and past) user jobs of:&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;CPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of CPUs used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;RAM&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the amount of RAM used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;GPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of GPUs used&lt;br /&gt;
#** note: 40 GB GPUs have double the impact on FairShare than 20 GB GPUs&lt;br /&gt;
# The impact on FairShare of any use of resources is &amp;#039;&amp;#039;&amp;#039;proportional to its duration&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
#* example: using 32 GB of RAM for 48h has the same impact of using 64 GB of RAM for 24h&lt;br /&gt;
# FairShare has a &amp;quot;fading memory&amp;quot;: i.e., resource use has less and less impact on FairShare the farther in the past it is&lt;br /&gt;
#* consequence: over time, the &amp;quot;history&amp;quot; of a user gets forgotten by FairShare&lt;br /&gt;
&lt;br /&gt;
== How to maximise the priority of your jobs ==&lt;br /&gt;
&lt;br /&gt;
Every time you run a SLURM job, follow these guidelines:&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Choose the less powerful QOS compatible with the needs of your job&lt;br /&gt;
:: QOSes with access to less resources have higher base priority&lt;br /&gt;
&lt;br /&gt;
; Only request CPUs and RAM that your job really needs&lt;br /&gt;
:: Usually, code exploits multiple CPUs only if specifically designed to do so: if unsure, only ask for 1 CPU&lt;br /&gt;
:: The fewer resources your job asks for, the less it will wait before they become available&lt;br /&gt;
:: Asking for fewer CPUs and/or less RAM improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Do not request more time than your jobs needs to complete&lt;br /&gt;
:: Make a worst-case estimate and only ask for that duration&lt;br /&gt;
:: Asking for less of Mufasa&amp;#039;s time improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Debug and test code using less powerful QOSes before running it with more powerful QOSes&lt;br /&gt;
:: Your test jobs will get a higher priority and your FairShare will improve&lt;br /&gt;
&lt;br /&gt;
; Cancel jobs that you don&amp;#039;t need them anymore&lt;br /&gt;
:: Always use [[User_Jobs#Cancelling_a_job_with_scancel|scancel]] to delete completed (or crashed) jobs: your Fairshare will improve&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Protip: it&amp;#039;s a good idea to [[User Jobs#Looking for unused GPUs|check for unused GPUs]] before choosing what to request. Requesting a GPU that is currently idle will help your job get executed sooner.&lt;br /&gt;
&lt;br /&gt;
= SLURM partitions =&lt;br /&gt;
&lt;br /&gt;
Partitions are another mechanism provided by SLURM to create different levels of access to system resources. Since in Mufasa 2.0 access to resources is controlled via [[#SLURM Quality of Service (QOS)|QOSes]], partitions are not very relevant. (This is one of the main differences between Mufasa 1.0 and Mufasa 2.0.)&lt;br /&gt;
&lt;br /&gt;
Note, however, that the default values for some features of SLURM jobs (e.g., duration) are [[#Default values|set by the partition]].&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, there is a single SLURM partition, called &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt;, and all jobs run on it. The state of &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt; can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sinfo -o &amp;quot;%10P %5a %9T %11L %10l&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PARTITION  AVAIL STATE     DEFAULTTIME TIMELIMIT &lt;br /&gt;
jobs*      up    idle      1:00:00     3-00:00:00&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where columns correspond to the following information:&lt;br /&gt;
&lt;br /&gt;
:; PARTITION&lt;br /&gt;
:: name of the partition; the asterisks indicates that it&amp;#039;s the default one&lt;br /&gt;
&lt;br /&gt;
:; AVAIL&lt;br /&gt;
:: state/availability of the partition: see [[#Partition availability|below]]&lt;br /&gt;
&lt;br /&gt;
:; STATE&lt;br /&gt;
:: state (using [https://slurm.schedmd.com/sinfo.html#SECTION_NODE-STATE-CODES these codes])&lt;br /&gt;
:: typical values are &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;mixed&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that some of the resources are busy executing jobs while other are idle, and &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;allocated&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that all of the resources are in use&lt;br /&gt;
&lt;br /&gt;
:; DEFAULTTIME&lt;br /&gt;
:: default runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:; TIMELIMIT&lt;br /&gt;
:: maximum runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
The asterisk at the end of the partition name indicates the default partition, i.e. the one on which jobs which do not ask for a specific partition are run.&lt;br /&gt;
&lt;br /&gt;
Command &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; does not tell you about the &amp;#039;&amp;#039;jobs&amp;#039;&amp;#039; submitted to a partition. This information is obtained, instead, with [[User Jobs#Inspecting jobs with squeue|command &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;]].&lt;br /&gt;
&lt;br /&gt;
== Partition availability ==&lt;br /&gt;
&lt;br /&gt;
The most important information that &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; provides is the &amp;#039;&amp;#039;&amp;#039;availability&amp;#039;&amp;#039;&amp;#039; (also called &amp;#039;&amp;#039;&amp;#039;state&amp;#039;&amp;#039;&amp;#039;) of partitions. This is shown in column &amp;quot;AVAIL&amp;quot;. Possible partition states are:&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is available&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed as soon as resources allow&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is in the process of becoming unavailable (i.e., of entering the &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; state: see below)&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is unavailable&lt;br /&gt;
:: There are no running jobs&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
When a partition goes from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; no harm is done to running jobs. In a normally functioning SLURM system, the passage from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; happens only when no jobs are running on the partition. If (e.g., due to a malfunction) the passage happens with jobs still running, they get killed.&lt;br /&gt;
&lt;br /&gt;
A partition in state &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; requires intervention by a [[Roles|Job Administrator]] to be restored to &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
== Default values ==&lt;br /&gt;
&lt;br /&gt;
The features of SLURM partitions, including the &amp;#039;&amp;#039;&amp;#039;default values&amp;#039;&amp;#039;&amp;#039; which are applied to jobs that do not make explicit requests, can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scontrol show partition&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PartitionName=jobs&lt;br /&gt;
   AllowGroups=ALL AllowAccounts=ALL AllowQos=nogpu,gpulight,gpu,gpuwide,gpuheavy-20,gpuheavy-40&lt;br /&gt;
   AllocNodes=ALL Default=YES QoS=N/A&lt;br /&gt;
=&amp;gt; DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO&lt;br /&gt;
   MaxNodes=UNLIMITED MaxTime=3-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED&lt;br /&gt;
   Nodes=gn01&lt;br /&gt;
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO&lt;br /&gt;
   OverTimeLimit=NONE PreemptMode=OFF&lt;br /&gt;
   State=UP TotalCPUs=48 TotalNodes=1 SelectTypeParameters=NONE&lt;br /&gt;
   JobDefaults=(null)&lt;br /&gt;
=&amp;gt; DefMemPerNode=4096 MaxMemPerNode=UNLIMITED&lt;br /&gt;
   TRES=cpu=48,mem=1011435M,node=1,billing=49,gres/gpu=13,gres/gpu:3g.20gb=5,gres/gpu:40gb=3,gres/gpu:4g.20gb=5&lt;br /&gt;
   TRESBillingWeights=cpu=1.0,gres/gpu:3g.20gb=6.0,gres/gpu:4g.20gb=6.0,gres/gpu:40gb=6.0,mem=0.05g&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In the example, we have highlighted with  &amp;#039;&amp;#039;&amp;#039;=&amp;gt;&amp;#039;&amp;#039;&amp;#039;  the most relevant default values for Mufasa users, i.e.:&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefaultTime&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default execution time assigned to a job run on the partition (e.g., 1 hour)&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefMemPerNode&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default amount of RAM assigned to a job run on the partition (e.g., 4GB)&lt;br /&gt;
&lt;br /&gt;
= System resources subjected to limitations =&lt;br /&gt;
&lt;br /&gt;
In systems based on SLURM like Mufasa, &amp;#039;&amp;#039;&amp;#039;TRES (Trackable RESources)&amp;#039;&amp;#039;&amp;#039; are (from [https://slurm.schedmd.com/tres.html SLURM&amp;#039;s documentation] &amp;quot;&amp;#039;&amp;#039;resources that can be tracked for usage or used to enforce limits against.&amp;#039;&amp;#039;&amp;quot;&lt;br /&gt;
&lt;br /&gt;
TRES include CPUs, RAM and &amp;#039;&amp;#039;&amp;#039;GRES&amp;#039;&amp;#039;&amp;#039;. The last term stands for &amp;#039;&amp;#039;Generic RESources&amp;#039;&amp;#039; that a job may need for its execution. In Mufasa, the only &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; resources are the GPUs.&lt;br /&gt;
&lt;br /&gt;
== &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax ==&lt;br /&gt;
&lt;br /&gt;
To ask SLURM to assign GRES resources (i.e., GPUs) to a job, a special syntax must be used. Precisely, the name of each GPU resource takes the form&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:Name:Type&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
Considering the [[System#CPUs and GPUs|GPU complement of Mufasa]], &amp;lt;code&amp;gt;Type&amp;lt;/code&amp;gt; takes the following values:&lt;br /&gt;
&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:40gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 40 Gbytes of RAM&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:4g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 4 compute units&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 3 compute units&lt;br /&gt;
&lt;br /&gt;
So, for instance,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
identifies a resource corresponding to a GPU with 20 GB of RAM and 3 compute units.&lt;br /&gt;
&lt;br /&gt;
When asking for a GRES resource (e.g., in an &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command or an &amp;lt;code&amp;gt;SBATCH&amp;lt;/code&amp;gt; directive of an [[User Jobs#Using execution scripts to run jobs|execution script]]), the syntax required by SLURM is&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:&amp;lt;Type&amp;gt;:&amp;lt;Quantity&amp;gt;&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
where &amp;lt;code&amp;gt;Quantity&amp;lt;/code&amp;gt; is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of type &amp;lt;code&amp;gt;4g.20gb&amp;lt;/code&amp;gt; the syntax is&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:4g.20gb:2&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SLURM&amp;#039;s &amp;#039;&amp;#039;generic resources&amp;#039;&amp;#039; are defined in &amp;lt;code&amp;gt;/etc/slurm/gres.conf&amp;lt;/code&amp;gt;. In order to make GPUs available to SLURM&amp;#039;s &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; management, Mufasa makes use of Nvidia&amp;#039;s [https://developer.nvidia.com/nvidia-management-library-nvml NVML library]. For additional information see [https://slurm.schedmd.com/gres.html SLURM&amp;#039;s documentation].&lt;/div&gt;</summary>
		<author><name>GiulioFontana</name></author>
	</entry>
	<entry>
		<id>https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2573</id>
		<title>SLURM</title>
		<link rel="alternate" type="text/html" href="https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2573"/>
		<updated>2026-11-03T14:37:52Z</updated>

		<summary type="html">&lt;p&gt;GiulioFontana: /* Elements determining job priority */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page presents the features of SLURM that are most relevant to Mufasa&amp;#039;s [[Roles|Job Users]]. Job Users can submit jobs for execution, cancel their own jobs, and see other users&amp;#039; jobs (but not intervene on them).&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa &amp;#039;&amp;#039;&amp;#039;must use SLURM&amp;#039;&amp;#039;&amp;#039; to run resource-heavy processes, i.e. computing jobs that require one or more of the following:&lt;br /&gt;
* GPUs&lt;br /&gt;
* multiple CPUs&lt;br /&gt;
* powerful CPUs&lt;br /&gt;
* a significant amount of RAM&lt;br /&gt;
&lt;br /&gt;
In fact, only processes run via SLURM have access to all the resources of Mufasa. Processes run outside SLURM are executed by the [[System#Login server|login server]] virtual machine, which has minimal resources and no GPUs. Using SLURM is therefore the only way to execute resource-heavy jobs on Mufasa (this is a key difference between Mufasa 1.0 and Mufasa 2.0).&lt;br /&gt;
&lt;br /&gt;
= SLURM in a nutshell =&lt;br /&gt;
&lt;br /&gt;
Computation jobs on Mufasa needs to be launched via [[System#The SLURM job scheduling system|SLURM]]. SLURM provides jobs with access to the [[#System resources subjected to limitations|physical resources]] of Mufasa, such as CPUs, GPUs and RAM. Thanks to SLURM, processing jobs share system resources, optimising their occupation and availability. &lt;br /&gt;
&lt;br /&gt;
When a user runs a job, the job does not get executed immediately and is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039;. SLURM executes jobs according to their order in the queue: the top job in the queue gets executed as soon as the necessary resources are available, while jobs lower in the queue wait longer. The position of a job in the queue is due to the &amp;#039;&amp;#039;&amp;#039;[[#Job priority|priority]]&amp;#039;&amp;#039;&amp;#039; assigned to it by SLURM, with higher-priority jobs closer to the top. As a general rule:&lt;br /&gt;
&lt;br /&gt;
;: &amp;#039;&amp;#039;&amp;#039;the greater the fraction of Mufasa&amp;#039;s overall resources that a job asks for, the lower the job&amp;#039;s priority will be&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
The priority mechanism is used to encourage users to use Mufasa&amp;#039;s resources (i.e.: GPUs, CPUs, RAM, execution time) in an effective and equitable manner. This page includes a [[#How_to_maximise_the_priority_of_your_jobs|chart explaining how to maximise the priority of your jobs]].&lt;br /&gt;
&lt;br /&gt;
The &amp;#039;&amp;#039;&amp;#039;time&amp;#039;&amp;#039;&amp;#039; available to a job for its execution is controlled by SLURM. When a user requests execution of a job, they must specify the duration of the time slot that the job needs. The job must complete its execution before the end of the requested time slot, because at the end of such slot it gets killed by SLURM.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0 access to system resources is managed via SLURM&amp;#039;s [[#SLURM Quality of Service (QOS)|Quality of Service (QOS)]] mechanism (Mufasa 1.0 used [[#SLURM_partitions|partitions]] instead). While [[User Jobs#Running jobs with SLURM|launching a processing job via SLURM]], the user must always specify the chosen QOS. QOSes differ in the set of resources that they provide access to because each of them is designed to fit a given type of job.&lt;br /&gt;
&lt;br /&gt;
Mufasa sets limits to the number of jobs by the same user. This page includes a [[#Limits on jobs by the same user|table summarising such limits]].&lt;br /&gt;
&lt;br /&gt;
= SLURM Quality of Service (QOS) =&lt;br /&gt;
&lt;br /&gt;
Through &amp;#039;&amp;#039;&amp;#039;Quality of Services&amp;#039;&amp;#039;&amp;#039; (&amp;#039;&amp;#039;&amp;#039;QOSes&amp;#039;&amp;#039;&amp;#039;), SLURM lets system configurators assign a name to a set of related constraints.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, QOSes are used to define different levels of access to the server&amp;#039;s resources. When [[User Jobs|executing a job with SLURM]], a user must always &amp;#039;&amp;#039;&amp;#039;specify the QOS&amp;#039;&amp;#039;&amp;#039; that their job will use: this choice, in turn, determines what resources the job is able to access and influences the [[#Job priority|priority]] of the job.&lt;br /&gt;
&lt;br /&gt;
Mufasa&amp;#039;s QOSes and their features can be inspected with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list qos format=name%-11,priority,MaxSubmit,maxwall,maxtres%-80,mintres%-18&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following (see below for the same data in table form):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
Name          Priority MaxSubmit     MaxWall MaxTRES                                                                          MinTRES            &lt;br /&gt;
----------- ---------- --------- ----------- -------------------------------------------------------------------------------- ------------------ &lt;br /&gt;
normal               0                                                                                                                           &lt;br /&gt;
nogpu                4         1  3-00:00:00 cpu=16,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=128G                    &lt;br /&gt;
gpuheavy-20          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=2,mem=128G             gres/gpu:4g.20gb=1 &lt;br /&gt;
gpuheavy-40          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=1,gres/gpu:4g.20gb=0,mem=128G             gres/gpu:40gb=1    &lt;br /&gt;
gpulight             8         1    12:00:00 cpu=2,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpu                  2         1  1-00:00:00 cpu=8,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpuwide              2         2  1-00:00:00 cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=1,mem=64G              gres/gpu:4g.20gb=1 &lt;br /&gt;
build               32         1    02:00:00 cpu=2,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=16G&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The columns of this output are the following:&lt;br /&gt;
&lt;br /&gt;
:; Name&lt;br /&gt;
:: name of the QOS&lt;br /&gt;
&lt;br /&gt;
:; Priority&lt;br /&gt;
:: priority tier associated to the QOS (higher value = higher priority): see [[#Job priority|Job priority]] for details&lt;br /&gt;
&lt;br /&gt;
:; MaxSubmit&lt;br /&gt;
:: maximum number of jobs from a single user that can be submitted to SLURM with this QOS; submitted jobs include both running and queued jobs&lt;br /&gt;
:: See [[#Limits on jobs by the same user|Limits on jobs by the same user]] for an overview of the limits on jobs set by Mufasa.&lt;br /&gt;
&lt;br /&gt;
:; MaxWall&lt;br /&gt;
:: maximum wall clock duration of the jobs using the QOS (after which they are killed by SLURM), in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
:: For some QOSes these are not set: it means that they are determined by the [[#SLURM partitions|partition]]. Partitions also define the [[#Default values|default duration]] of jobs.&lt;br /&gt;
&lt;br /&gt;
:; MaxTRES&lt;br /&gt;
:: amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) available to a job using the QOS, where&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;cpu=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of CPUs (i.e., processor cores) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of CPUs specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;gres/&amp;#039;&amp;#039;gpu:Type&amp;#039;&amp;#039;=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of GPUs of class &amp;lt;code&amp;gt;&amp;#039;&amp;#039;Type&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; (see [[User Jobs#gres syntax|&amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax]]) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; (for QOSes that allow access to GPUs) if not specified, the job cannot be launched&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;mem=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;G&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum amount of system RAM is &amp;#039;&amp;#039;K&amp;#039;&amp;#039; GBytes&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of RAM specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
&lt;br /&gt;
:; MinTRES&lt;br /&gt;
:: minimum amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) that a job using the QOS must request in order to actually get executed by SLURM.&lt;br /&gt;
:: (If your job does not actually need these resources, you&amp;#039;ve chosen the wrong QOS: you can use one with a higher priority.)&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; QOS is the default one, and exists only to ensure that users always specify a QOS when running a job. Since &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; has zero priority and no resources, a job run using this QOS would never be run.&lt;br /&gt;
&lt;br /&gt;
The information provided by the &amp;lt;code&amp;gt;sacctmgr list qos&amp;lt;/code&amp;gt; command above is summarised by the following table:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! QOS&lt;br /&gt;
! Priority&amp;lt;/br&amp;gt;tier&lt;br /&gt;
! Max&amp;lt;/br&amp;gt;Submit&lt;br /&gt;
! MaxWall&amp;lt;/br&amp;gt;[h]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;CPUs&lt;br /&gt;
! max RAM&amp;lt;/br&amp;gt;[GB]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;3g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;4g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;40GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | build&lt;br /&gt;
| 32&lt;br /&gt;
| 1&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 16&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpulight&lt;br /&gt;
| 8&lt;br /&gt;
| 1&lt;br /&gt;
| 12&lt;br /&gt;
| 2&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | nogpu&lt;br /&gt;
| 4&lt;br /&gt;
| 1&lt;br /&gt;
| 72&lt;br /&gt;
| 16&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpu&lt;br /&gt;
| 2&lt;br /&gt;
| 1&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuwide&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-20&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| 2 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-40&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
A key piece of information in the table above is the &amp;#039;&amp;#039;&amp;#039;priority tier&amp;#039;&amp;#039;&amp;#039; associated to each QOS. In Mufasa 2.0, priority tiers are used to encourage users to use the &amp;#039;&amp;#039;&amp;#039;least powerful QOS that is compatible with their needs&amp;#039;&amp;#039;&amp;#039;, where &amp;quot;powerful&amp;quot; means &amp;quot;rich with resources&amp;quot;. Less powerful QOSes increase the priority of the jobs that use them, so these jobs tend to be executed sooner. See [[#Job priority|Job priority]] for details about how priority affects the execution order of jobs in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
== The &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS ==&lt;br /&gt;
&lt;br /&gt;
This QOS is specifically designed to be used by Mufasa users to &amp;#039;&amp;#039;&amp;#039;quickly build [[System#Containers|container images]]&amp;#039;&amp;#039;&amp;#039;. Its associated priority tier is very high: this, combined with the fact that jobs using this QOS require few resources, means that such jobs usually get executed very soon.&lt;br /&gt;
&lt;br /&gt;
On the other hand, the limited resources and the lack of access to the GPUs make the &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS unsuitable for other tasks.&lt;br /&gt;
&lt;br /&gt;
See [[Singularity#Building Singularity images|Building Singularity images]] for directions about building Singularity container images.&lt;br /&gt;
&lt;br /&gt;
== Restricted QOSes ==&lt;br /&gt;
&lt;br /&gt;
In Mufasa, the most powerful QOSes are reserved to researchers (called &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users: these include academic personnel and Ph.D. students). M.Sc. students, called &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users, cannot use them while running jobs.&lt;br /&gt;
&lt;br /&gt;
See [[#research users and students users|below]] to understand how &amp;#039;&amp;#039;user categories&amp;#039;&amp;#039; work in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
= &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users =&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa belong to two &amp;#039;&amp;#039;&amp;#039;user categories&amp;#039;&amp;#039;&amp;#039;. Categories provide users with different access to  to Mufasa&amp;#039;s resources: the idea being to provide researchers with more access while still letting students benefit from the server&amp;#039;s capabilities.&lt;br /&gt;
&lt;br /&gt;
User categories in Mufasa are the following:&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. academic personnel and Ph.D. students&lt;br /&gt;
::: * can use all QOSes, including the the [[#Restricted QOSes|restricted ones]]&lt;br /&gt;
::: * their jobs have a higher &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a higher number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. M.Sc. students&lt;br /&gt;
::: * cannot use the [[#Restricted QOSes|restricted QOSes]]&lt;br /&gt;
::: * their jobs have a lower &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a lower number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
You can inspect the differences in priority and running jobs between &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association format=&amp;quot;account,priority,maxjobs&amp;quot; | grep -E &amp;#039;Account|research|students&amp;#039;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
   Account   Priority MaxJobs &lt;br /&gt;
  research          4       2 &lt;br /&gt;
  students          1       1&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
To know what limits apply to your own user, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association where user=$USER format=&amp;quot;user,priority,maxjobs,qos%-60&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
      User   Priority MaxJobs QOS                                                          &lt;br /&gt;
---------- ---------- ------- ------------------------------------------------------------ &lt;br /&gt;
    preali          4       2 build,gpu,gpuheavy-20,gpuheavy-40,gpulight,gpuwide,nogpu&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The list under &amp;quot;QOS&amp;quot; shows what QOSes your user is allowed to use when [[User Jobs|running jobs]]. &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users can use all of them, while &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users can only access a subset of them.&lt;br /&gt;
&lt;br /&gt;
= Limits on jobs by the same user =&lt;br /&gt;
&lt;br /&gt;
Mufasa enforces limits on the number of jobs from a single user. Such limits aim at preventing users from &amp;quot;hogging&amp;quot; system resources, and apply to:&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;submitted jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that the user asked SLURM to execute, each of which may currently be either running or queued&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;running jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that are currently in execution&lt;br /&gt;
&lt;br /&gt;
Limits on submitted jobs are set via [[#SLURM Quality of Service (QOS)|QOSes]], while limits on running jobs are set via [[#research users and students users|user category]].&lt;br /&gt;
&lt;br /&gt;
The following table summarises what limits exist:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! &lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;running&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;submitted&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | global limits&amp;lt;br/&amp;gt;(system-wide)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to limits set by individual QOSes (below)&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | limits for&amp;lt;br/&amp;gt;individual&amp;lt;/br&amp;gt;QOSes&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to global limits (above)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for the&amp;#039;&amp;#039; &amp;lt;code&amp;gt;gpuwide&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;QOS&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for each of the other QOSes&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
= Job priority =&lt;br /&gt;
&lt;br /&gt;
Once the execution of a job has been requested, the job is not run immediately: it is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039; by SLURM, together with all the other jobs awaiting execution. The job on top of the queue at any time is the first to be put into execution as soon as the resources it requires are available. &lt;br /&gt;
&lt;br /&gt;
The order of the items in the job queue depends on their &amp;#039;&amp;#039;&amp;#039;priority&amp;#039;&amp;#039;&amp;#039;. The job with the highest priority is put by SLURM on top of the queue, while all other jobs are put in the queue in order of descending priority.&lt;br /&gt;
&lt;br /&gt;
The goal of SLURM is to maximise resource availability: i.e., to ensure the shorter possible wait time before job execution. To achieve this goal, in Mufasa SLURM is configured to &amp;#039;&amp;#039;&amp;#039;encourage users not to ask for resources or execution time that their job doesn&amp;#039;t need&amp;#039;&amp;#039;&amp;#039;. This is done via the priority mechanism: the more resources and/or time a job requests, the lower its priority will be; and the later it will be executed.&lt;br /&gt;
&lt;br /&gt;
Priority management in Mufasa is designed to set up a &amp;#039;&amp;#039;&amp;#039;virtuous cycle&amp;#039;&amp;#039;&amp;#039; where users, by carefully choosing what to ask for, obtain two results:&lt;br /&gt;
* they ensure that their job is executed as soon as possible;&lt;br /&gt;
* they leave as much as possible of Mufasa&amp;#039;s resources free for other users&amp;#039;s jobs.&lt;br /&gt;
&lt;br /&gt;
== Elements determining job priority ==&lt;br /&gt;
In Mufasa, the priority of a job is computed by SLURM according to the following elements:&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#research users and students users|User category]]&amp;#039;&amp;#039;&amp;#039; (i.e., &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;)&lt;br /&gt;
::: Used to provide higher priority to jobs run by &amp;#039;&amp;#039;&amp;#039;research personnel&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#SLURM Quality of Service (QOS)|QOS]]&amp;#039;&amp;#039;&amp;#039; used by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;less resources&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Number of CPUs&amp;#039;&amp;#039;&amp;#039; requested by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;a lower number of CPUs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job Age&amp;#039;&amp;#039;&amp;#039;, i.e. the time that the job has been waiting in the queue&lt;br /&gt;
::: Used to provide higher priority to jobs which have been &amp;#039;&amp;#039;&amp;#039;queued for a longer time&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;FairShare&amp;#039;&amp;#039;&amp;#039;, i.e. a factor computed by SLURM to balance use of the system by different users&lt;br /&gt;
::: Used to provide higher priority to jobs by users who &amp;#039;&amp;#039;&amp;#039;used Mufasa&amp;#039;s resources less than others&amp;#039;&amp;#039;&amp;#039; (see [[#How FairShare works|below]])&lt;br /&gt;
&lt;br /&gt;
You can understand how each of these elements contribute to the priority of your jobs waiting for execution with command &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;. A tailored version of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; that only displays the parameters listed above is the following:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sprio -o &amp;quot;%6i %9u %.5Y %.10B %.5Q %.9T %.5A %.5F&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
; Important note!&lt;br /&gt;
:: The output of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; is only relevant for jobs &amp;#039;&amp;#039;&amp;#039;not yet in execution&amp;#039;&amp;#039;&amp;#039;. For an already running job, the value shown by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; is the priority that the job would get if still queued, which usually does not correspond to its actual priority.&lt;br /&gt;
&lt;br /&gt;
Here is an example of its output:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
JOBID  USER      PRIOR      ASSOC   QOS      TRES   AGE FAIRS&lt;br /&gt;
5774   itombini   3492       2500   750  cpu=-800   906   136&lt;br /&gt;
5780   fbottazzi  2266        625   750  cpu=-600   420  1071&lt;br /&gt;
5783   mzoccola   1766        625   750  cpu=-800   217   974&lt;br /&gt;
5790   amoglia    2717       2500   375  cpu=-800   156   487&lt;br /&gt;
5791   qliu       2232       2500   375  cpu=-800   138    19&lt;br /&gt;
5792   fpastori   2197       2500   375  cpu=-800    84    39&lt;br /&gt;
5793   bagostini  2034        625  1500 cpu=-1200    18  1091&lt;br /&gt;
5794   lpassoni   2517       2500   750  cpu=-800     9    58&lt;br /&gt;
5795   lpassoni   2517       2500   750  cpu=-800     9    58&lt;br /&gt;
5796   adurat     3101       2500   750  cpu=-800     8   643&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
In it, column ASSOC (i.e., association) corresponds to user category, while column TRES corresponds to the number of CPUs. Note that the contribution of CPUs to priority is &amp;#039;&amp;#039;negative&amp;#039;&amp;#039;, i.e. it is a penalty. Column PRIOR provides the overall priority of each job.&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
=== How FairShare works ===&lt;br /&gt;
In Mufasa 2.0, the FairShare of each user is computed according to these rules:&lt;br /&gt;
# It considers the quantity used by (current and past) user jobs of:&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;CPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of CPUs used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;RAM&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the amount of RAM used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;GPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of GPUs used&lt;br /&gt;
#** note: 40 GB GPUs have double the impact on FairShare than 20 GB GPUs&lt;br /&gt;
# The impact on FairShare of any use of resources is &amp;#039;&amp;#039;&amp;#039;proportional to its duration&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
#* example: using 32 GB of RAM for 48h has the same impact of using 64 GB of RAM for 24h&lt;br /&gt;
# FairShare has a &amp;quot;fading memory&amp;quot;: i.e., resource use has less and less impact on FairShare the farther in the past it is&lt;br /&gt;
#* consequence: over time, the &amp;quot;history&amp;quot; of a user gets forgotten by FairShare&lt;br /&gt;
&lt;br /&gt;
== How to maximise the priority of your jobs ==&lt;br /&gt;
&lt;br /&gt;
Every time you run a SLURM job, follow these guidelines:&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Choose the less powerful QOS compatible with the needs of your job&lt;br /&gt;
:: QOSes with access to less resources have higher base priority&lt;br /&gt;
&lt;br /&gt;
; Only request CPUs and RAM that your job really needs&lt;br /&gt;
:: Usually, code exploits multiple CPUs only if specifically designed to do so: if unsure, only ask for 1 CPU&lt;br /&gt;
:: The fewer resources your job asks for, the less it will wait before they become available&lt;br /&gt;
:: Asking for fewer CPUs and/or less RAM improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Do not request more time than your jobs needs to complete&lt;br /&gt;
:: Make a worst-case estimate and only ask for that duration&lt;br /&gt;
:: Asking for less of Mufasa&amp;#039;s time improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Debug and test code using less powerful QOSes before running it with more powerful QOSes&lt;br /&gt;
:: Your test jobs will get a higher priority and your FairShare will improve&lt;br /&gt;
&lt;br /&gt;
; Cancel jobs that you don&amp;#039;t need them anymore&lt;br /&gt;
:: Always use [[User_Jobs#Cancelling_a_job_with_scancel|scancel]] to delete completed (or crashed) jobs: your Fairshare will improve&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Protip: it&amp;#039;s a good idea to [[User Jobs#Looking for unused GPUs|check for unused GPUs]] before choosing what to request. Requesting a GPU that is currently idle will help your job get executed sooner.&lt;br /&gt;
&lt;br /&gt;
= SLURM partitions =&lt;br /&gt;
&lt;br /&gt;
Partitions are another mechanism provided by SLURM to create different levels of access to system resources. Since in Mufasa 2.0 access to resources is controlled via [[#SLURM Quality of Service (QOS)|QOSes]], partitions are not very relevant. (This is one of the main differences between Mufasa 1.0 and Mufasa 2.0.)&lt;br /&gt;
&lt;br /&gt;
Note, however, that the default values for some features of SLURM jobs (e.g., duration) are [[#Default values|set by the partition]].&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, there is a single SLURM partition, called &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt;, and all jobs run on it. The state of &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt; can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sinfo -o &amp;quot;%10P %5a %9T %11L %10l&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PARTITION  AVAIL STATE     DEFAULTTIME TIMELIMIT &lt;br /&gt;
jobs*      up    idle      1:00:00     3-00:00:00&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where columns correspond to the following information:&lt;br /&gt;
&lt;br /&gt;
:; PARTITION&lt;br /&gt;
:: name of the partition; the asterisks indicates that it&amp;#039;s the default one&lt;br /&gt;
&lt;br /&gt;
:; AVAIL&lt;br /&gt;
:: state/availability of the partition: see [[#Partition availability|below]]&lt;br /&gt;
&lt;br /&gt;
:; STATE&lt;br /&gt;
:: state (using [https://slurm.schedmd.com/sinfo.html#SECTION_NODE-STATE-CODES these codes])&lt;br /&gt;
:: typical values are &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;mixed&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that some of the resources are busy executing jobs while other are idle, and &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;allocated&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that all of the resources are in use&lt;br /&gt;
&lt;br /&gt;
:; DEFAULTTIME&lt;br /&gt;
:: default runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:; TIMELIMIT&lt;br /&gt;
:: maximum runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
The asterisk at the end of the partition name indicates the default partition, i.e. the one on which jobs which do not ask for a specific partition are run.&lt;br /&gt;
&lt;br /&gt;
Command &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; does not tell you about the &amp;#039;&amp;#039;jobs&amp;#039;&amp;#039; submitted to a partition. This information is obtained, instead, with [[User Jobs#Inspecting jobs with squeue|command &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;]].&lt;br /&gt;
&lt;br /&gt;
== Partition availability ==&lt;br /&gt;
&lt;br /&gt;
The most important information that &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; provides is the &amp;#039;&amp;#039;&amp;#039;availability&amp;#039;&amp;#039;&amp;#039; (also called &amp;#039;&amp;#039;&amp;#039;state&amp;#039;&amp;#039;&amp;#039;) of partitions. This is shown in column &amp;quot;AVAIL&amp;quot;. Possible partition states are:&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is available&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed as soon as resources allow&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is in the process of becoming unavailable (i.e., of entering the &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; state: see below)&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is unavailable&lt;br /&gt;
:: There are no running jobs&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
When a partition goes from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; no harm is done to running jobs. In a normally functioning SLURM system, the passage from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; happens only when no jobs are running on the partition. If (e.g., due to a malfunction) the passage happens with jobs still running, they get killed.&lt;br /&gt;
&lt;br /&gt;
A partition in state &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; requires intervention by a [[Roles|Job Administrator]] to be restored to &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
== Default values ==&lt;br /&gt;
&lt;br /&gt;
The features of SLURM partitions, including the &amp;#039;&amp;#039;&amp;#039;default values&amp;#039;&amp;#039;&amp;#039; which are applied to jobs that do not make explicit requests, can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scontrol show partition&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PartitionName=jobs&lt;br /&gt;
   AllowGroups=ALL AllowAccounts=ALL AllowQos=nogpu,gpulight,gpu,gpuwide,gpuheavy-20,gpuheavy-40&lt;br /&gt;
   AllocNodes=ALL Default=YES QoS=N/A&lt;br /&gt;
=&amp;gt; DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO&lt;br /&gt;
   MaxNodes=UNLIMITED MaxTime=3-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED&lt;br /&gt;
   Nodes=gn01&lt;br /&gt;
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO&lt;br /&gt;
   OverTimeLimit=NONE PreemptMode=OFF&lt;br /&gt;
   State=UP TotalCPUs=48 TotalNodes=1 SelectTypeParameters=NONE&lt;br /&gt;
   JobDefaults=(null)&lt;br /&gt;
=&amp;gt; DefMemPerNode=4096 MaxMemPerNode=UNLIMITED&lt;br /&gt;
   TRES=cpu=48,mem=1011435M,node=1,billing=49,gres/gpu=13,gres/gpu:3g.20gb=5,gres/gpu:40gb=3,gres/gpu:4g.20gb=5&lt;br /&gt;
   TRESBillingWeights=cpu=1.0,gres/gpu:3g.20gb=6.0,gres/gpu:4g.20gb=6.0,gres/gpu:40gb=6.0,mem=0.05g&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In the example, we have highlighted with  &amp;#039;&amp;#039;&amp;#039;=&amp;gt;&amp;#039;&amp;#039;&amp;#039;  the most relevant default values for Mufasa users, i.e.:&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefaultTime&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default execution time assigned to a job run on the partition (e.g., 1 hour)&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefMemPerNode&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default amount of RAM assigned to a job run on the partition (e.g., 4GB)&lt;br /&gt;
&lt;br /&gt;
= System resources subjected to limitations =&lt;br /&gt;
&lt;br /&gt;
In systems based on SLURM like Mufasa, &amp;#039;&amp;#039;&amp;#039;TRES (Trackable RESources)&amp;#039;&amp;#039;&amp;#039; are (from [https://slurm.schedmd.com/tres.html SLURM&amp;#039;s documentation] &amp;quot;&amp;#039;&amp;#039;resources that can be tracked for usage or used to enforce limits against.&amp;#039;&amp;#039;&amp;quot;&lt;br /&gt;
&lt;br /&gt;
TRES include CPUs, RAM and &amp;#039;&amp;#039;&amp;#039;GRES&amp;#039;&amp;#039;&amp;#039;. The last term stands for &amp;#039;&amp;#039;Generic RESources&amp;#039;&amp;#039; that a job may need for its execution. In Mufasa, the only &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; resources are the GPUs.&lt;br /&gt;
&lt;br /&gt;
== &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax ==&lt;br /&gt;
&lt;br /&gt;
To ask SLURM to assign GRES resources (i.e., GPUs) to a job, a special syntax must be used. Precisely, the name of each GPU resource takes the form&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:Name:Type&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
Considering the [[System#CPUs and GPUs|GPU complement of Mufasa]], &amp;lt;code&amp;gt;Type&amp;lt;/code&amp;gt; takes the following values:&lt;br /&gt;
&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:40gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 40 Gbytes of RAM&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:4g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 4 compute units&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 3 compute units&lt;br /&gt;
&lt;br /&gt;
So, for instance,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
identifies a resource corresponding to a GPU with 20 GB of RAM and 3 compute units.&lt;br /&gt;
&lt;br /&gt;
When asking for a GRES resource (e.g., in an &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command or an &amp;lt;code&amp;gt;SBATCH&amp;lt;/code&amp;gt; directive of an [[User Jobs#Using execution scripts to run jobs|execution script]]), the syntax required by SLURM is&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:&amp;lt;Type&amp;gt;:&amp;lt;Quantity&amp;gt;&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
where &amp;lt;code&amp;gt;Quantity&amp;lt;/code&amp;gt; is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of type &amp;lt;code&amp;gt;4g.20gb&amp;lt;/code&amp;gt; the syntax is&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:4g.20gb:2&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SLURM&amp;#039;s &amp;#039;&amp;#039;generic resources&amp;#039;&amp;#039; are defined in &amp;lt;code&amp;gt;/etc/slurm/gres.conf&amp;lt;/code&amp;gt;. In order to make GPUs available to SLURM&amp;#039;s &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; management, Mufasa makes use of Nvidia&amp;#039;s [https://developer.nvidia.com/nvidia-management-library-nvml NVML library]. For additional information see [https://slurm.schedmd.com/gres.html SLURM&amp;#039;s documentation].&lt;/div&gt;</summary>
		<author><name>GiulioFontana</name></author>
	</entry>
	<entry>
		<id>https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2572</id>
		<title>SLURM</title>
		<link rel="alternate" type="text/html" href="https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2572"/>
		<updated>2026-11-03T14:36:57Z</updated>

		<summary type="html">&lt;p&gt;GiulioFontana: /* Elements determining job priority */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page presents the features of SLURM that are most relevant to Mufasa&amp;#039;s [[Roles|Job Users]]. Job Users can submit jobs for execution, cancel their own jobs, and see other users&amp;#039; jobs (but not intervene on them).&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa &amp;#039;&amp;#039;&amp;#039;must use SLURM&amp;#039;&amp;#039;&amp;#039; to run resource-heavy processes, i.e. computing jobs that require one or more of the following:&lt;br /&gt;
* GPUs&lt;br /&gt;
* multiple CPUs&lt;br /&gt;
* powerful CPUs&lt;br /&gt;
* a significant amount of RAM&lt;br /&gt;
&lt;br /&gt;
In fact, only processes run via SLURM have access to all the resources of Mufasa. Processes run outside SLURM are executed by the [[System#Login server|login server]] virtual machine, which has minimal resources and no GPUs. Using SLURM is therefore the only way to execute resource-heavy jobs on Mufasa (this is a key difference between Mufasa 1.0 and Mufasa 2.0).&lt;br /&gt;
&lt;br /&gt;
= SLURM in a nutshell =&lt;br /&gt;
&lt;br /&gt;
Computation jobs on Mufasa needs to be launched via [[System#The SLURM job scheduling system|SLURM]]. SLURM provides jobs with access to the [[#System resources subjected to limitations|physical resources]] of Mufasa, such as CPUs, GPUs and RAM. Thanks to SLURM, processing jobs share system resources, optimising their occupation and availability. &lt;br /&gt;
&lt;br /&gt;
When a user runs a job, the job does not get executed immediately and is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039;. SLURM executes jobs according to their order in the queue: the top job in the queue gets executed as soon as the necessary resources are available, while jobs lower in the queue wait longer. The position of a job in the queue is due to the &amp;#039;&amp;#039;&amp;#039;[[#Job priority|priority]]&amp;#039;&amp;#039;&amp;#039; assigned to it by SLURM, with higher-priority jobs closer to the top. As a general rule:&lt;br /&gt;
&lt;br /&gt;
;: &amp;#039;&amp;#039;&amp;#039;the greater the fraction of Mufasa&amp;#039;s overall resources that a job asks for, the lower the job&amp;#039;s priority will be&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
The priority mechanism is used to encourage users to use Mufasa&amp;#039;s resources (i.e.: GPUs, CPUs, RAM, execution time) in an effective and equitable manner. This page includes a [[#How_to_maximise_the_priority_of_your_jobs|chart explaining how to maximise the priority of your jobs]].&lt;br /&gt;
&lt;br /&gt;
The &amp;#039;&amp;#039;&amp;#039;time&amp;#039;&amp;#039;&amp;#039; available to a job for its execution is controlled by SLURM. When a user requests execution of a job, they must specify the duration of the time slot that the job needs. The job must complete its execution before the end of the requested time slot, because at the end of such slot it gets killed by SLURM.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0 access to system resources is managed via SLURM&amp;#039;s [[#SLURM Quality of Service (QOS)|Quality of Service (QOS)]] mechanism (Mufasa 1.0 used [[#SLURM_partitions|partitions]] instead). While [[User Jobs#Running jobs with SLURM|launching a processing job via SLURM]], the user must always specify the chosen QOS. QOSes differ in the set of resources that they provide access to because each of them is designed to fit a given type of job.&lt;br /&gt;
&lt;br /&gt;
Mufasa sets limits to the number of jobs by the same user. This page includes a [[#Limits on jobs by the same user|table summarising such limits]].&lt;br /&gt;
&lt;br /&gt;
= SLURM Quality of Service (QOS) =&lt;br /&gt;
&lt;br /&gt;
Through &amp;#039;&amp;#039;&amp;#039;Quality of Services&amp;#039;&amp;#039;&amp;#039; (&amp;#039;&amp;#039;&amp;#039;QOSes&amp;#039;&amp;#039;&amp;#039;), SLURM lets system configurators assign a name to a set of related constraints.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, QOSes are used to define different levels of access to the server&amp;#039;s resources. When [[User Jobs|executing a job with SLURM]], a user must always &amp;#039;&amp;#039;&amp;#039;specify the QOS&amp;#039;&amp;#039;&amp;#039; that their job will use: this choice, in turn, determines what resources the job is able to access and influences the [[#Job priority|priority]] of the job.&lt;br /&gt;
&lt;br /&gt;
Mufasa&amp;#039;s QOSes and their features can be inspected with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list qos format=name%-11,priority,MaxSubmit,maxwall,maxtres%-80,mintres%-18&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following (see below for the same data in table form):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
Name          Priority MaxSubmit     MaxWall MaxTRES                                                                          MinTRES            &lt;br /&gt;
----------- ---------- --------- ----------- -------------------------------------------------------------------------------- ------------------ &lt;br /&gt;
normal               0                                                                                                                           &lt;br /&gt;
nogpu                4         1  3-00:00:00 cpu=16,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=128G                    &lt;br /&gt;
gpuheavy-20          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=2,mem=128G             gres/gpu:4g.20gb=1 &lt;br /&gt;
gpuheavy-40          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=1,gres/gpu:4g.20gb=0,mem=128G             gres/gpu:40gb=1    &lt;br /&gt;
gpulight             8         1    12:00:00 cpu=2,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpu                  2         1  1-00:00:00 cpu=8,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpuwide              2         2  1-00:00:00 cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=1,mem=64G              gres/gpu:4g.20gb=1 &lt;br /&gt;
build               32         1    02:00:00 cpu=2,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=16G&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The columns of this output are the following:&lt;br /&gt;
&lt;br /&gt;
:; Name&lt;br /&gt;
:: name of the QOS&lt;br /&gt;
&lt;br /&gt;
:; Priority&lt;br /&gt;
:: priority tier associated to the QOS (higher value = higher priority): see [[#Job priority|Job priority]] for details&lt;br /&gt;
&lt;br /&gt;
:; MaxSubmit&lt;br /&gt;
:: maximum number of jobs from a single user that can be submitted to SLURM with this QOS; submitted jobs include both running and queued jobs&lt;br /&gt;
:: See [[#Limits on jobs by the same user|Limits on jobs by the same user]] for an overview of the limits on jobs set by Mufasa.&lt;br /&gt;
&lt;br /&gt;
:; MaxWall&lt;br /&gt;
:: maximum wall clock duration of the jobs using the QOS (after which they are killed by SLURM), in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
:: For some QOSes these are not set: it means that they are determined by the [[#SLURM partitions|partition]]. Partitions also define the [[#Default values|default duration]] of jobs.&lt;br /&gt;
&lt;br /&gt;
:; MaxTRES&lt;br /&gt;
:: amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) available to a job using the QOS, where&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;cpu=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of CPUs (i.e., processor cores) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of CPUs specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;gres/&amp;#039;&amp;#039;gpu:Type&amp;#039;&amp;#039;=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of GPUs of class &amp;lt;code&amp;gt;&amp;#039;&amp;#039;Type&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; (see [[User Jobs#gres syntax|&amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax]]) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; (for QOSes that allow access to GPUs) if not specified, the job cannot be launched&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;mem=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;G&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum amount of system RAM is &amp;#039;&amp;#039;K&amp;#039;&amp;#039; GBytes&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of RAM specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
&lt;br /&gt;
:; MinTRES&lt;br /&gt;
:: minimum amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) that a job using the QOS must request in order to actually get executed by SLURM.&lt;br /&gt;
:: (If your job does not actually need these resources, you&amp;#039;ve chosen the wrong QOS: you can use one with a higher priority.)&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; QOS is the default one, and exists only to ensure that users always specify a QOS when running a job. Since &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; has zero priority and no resources, a job run using this QOS would never be run.&lt;br /&gt;
&lt;br /&gt;
The information provided by the &amp;lt;code&amp;gt;sacctmgr list qos&amp;lt;/code&amp;gt; command above is summarised by the following table:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! QOS&lt;br /&gt;
! Priority&amp;lt;/br&amp;gt;tier&lt;br /&gt;
! Max&amp;lt;/br&amp;gt;Submit&lt;br /&gt;
! MaxWall&amp;lt;/br&amp;gt;[h]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;CPUs&lt;br /&gt;
! max RAM&amp;lt;/br&amp;gt;[GB]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;3g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;4g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;40GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | build&lt;br /&gt;
| 32&lt;br /&gt;
| 1&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 16&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpulight&lt;br /&gt;
| 8&lt;br /&gt;
| 1&lt;br /&gt;
| 12&lt;br /&gt;
| 2&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | nogpu&lt;br /&gt;
| 4&lt;br /&gt;
| 1&lt;br /&gt;
| 72&lt;br /&gt;
| 16&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpu&lt;br /&gt;
| 2&lt;br /&gt;
| 1&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuwide&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-20&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| 2 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-40&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
A key piece of information in the table above is the &amp;#039;&amp;#039;&amp;#039;priority tier&amp;#039;&amp;#039;&amp;#039; associated to each QOS. In Mufasa 2.0, priority tiers are used to encourage users to use the &amp;#039;&amp;#039;&amp;#039;least powerful QOS that is compatible with their needs&amp;#039;&amp;#039;&amp;#039;, where &amp;quot;powerful&amp;quot; means &amp;quot;rich with resources&amp;quot;. Less powerful QOSes increase the priority of the jobs that use them, so these jobs tend to be executed sooner. See [[#Job priority|Job priority]] for details about how priority affects the execution order of jobs in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
== The &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS ==&lt;br /&gt;
&lt;br /&gt;
This QOS is specifically designed to be used by Mufasa users to &amp;#039;&amp;#039;&amp;#039;quickly build [[System#Containers|container images]]&amp;#039;&amp;#039;&amp;#039;. Its associated priority tier is very high: this, combined with the fact that jobs using this QOS require few resources, means that such jobs usually get executed very soon.&lt;br /&gt;
&lt;br /&gt;
On the other hand, the limited resources and the lack of access to the GPUs make the &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS unsuitable for other tasks.&lt;br /&gt;
&lt;br /&gt;
See [[Singularity#Building Singularity images|Building Singularity images]] for directions about building Singularity container images.&lt;br /&gt;
&lt;br /&gt;
== Restricted QOSes ==&lt;br /&gt;
&lt;br /&gt;
In Mufasa, the most powerful QOSes are reserved to researchers (called &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users: these include academic personnel and Ph.D. students). M.Sc. students, called &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users, cannot use them while running jobs.&lt;br /&gt;
&lt;br /&gt;
See [[#research users and students users|below]] to understand how &amp;#039;&amp;#039;user categories&amp;#039;&amp;#039; work in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
= &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users =&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa belong to two &amp;#039;&amp;#039;&amp;#039;user categories&amp;#039;&amp;#039;&amp;#039;. Categories provide users with different access to  to Mufasa&amp;#039;s resources: the idea being to provide researchers with more access while still letting students benefit from the server&amp;#039;s capabilities.&lt;br /&gt;
&lt;br /&gt;
User categories in Mufasa are the following:&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. academic personnel and Ph.D. students&lt;br /&gt;
::: * can use all QOSes, including the the [[#Restricted QOSes|restricted ones]]&lt;br /&gt;
::: * their jobs have a higher &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a higher number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. M.Sc. students&lt;br /&gt;
::: * cannot use the [[#Restricted QOSes|restricted QOSes]]&lt;br /&gt;
::: * their jobs have a lower &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a lower number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
You can inspect the differences in priority and running jobs between &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association format=&amp;quot;account,priority,maxjobs&amp;quot; | grep -E &amp;#039;Account|research|students&amp;#039;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
   Account   Priority MaxJobs &lt;br /&gt;
  research          4       2 &lt;br /&gt;
  students          1       1&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
To know what limits apply to your own user, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association where user=$USER format=&amp;quot;user,priority,maxjobs,qos%-60&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
      User   Priority MaxJobs QOS                                                          &lt;br /&gt;
---------- ---------- ------- ------------------------------------------------------------ &lt;br /&gt;
    preali          4       2 build,gpu,gpuheavy-20,gpuheavy-40,gpulight,gpuwide,nogpu&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The list under &amp;quot;QOS&amp;quot; shows what QOSes your user is allowed to use when [[User Jobs|running jobs]]. &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users can use all of them, while &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users can only access a subset of them.&lt;br /&gt;
&lt;br /&gt;
= Limits on jobs by the same user =&lt;br /&gt;
&lt;br /&gt;
Mufasa enforces limits on the number of jobs from a single user. Such limits aim at preventing users from &amp;quot;hogging&amp;quot; system resources, and apply to:&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;submitted jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that the user asked SLURM to execute, each of which may currently be either running or queued&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;running jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that are currently in execution&lt;br /&gt;
&lt;br /&gt;
Limits on submitted jobs are set via [[#SLURM Quality of Service (QOS)|QOSes]], while limits on running jobs are set via [[#research users and students users|user category]].&lt;br /&gt;
&lt;br /&gt;
The following table summarises what limits exist:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! &lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;running&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;submitted&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | global limits&amp;lt;br/&amp;gt;(system-wide)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to limits set by individual QOSes (below)&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | limits for&amp;lt;br/&amp;gt;individual&amp;lt;/br&amp;gt;QOSes&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to global limits (above)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for the&amp;#039;&amp;#039; &amp;lt;code&amp;gt;gpuwide&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;QOS&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for each of the other QOSes&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
= Job priority =&lt;br /&gt;
&lt;br /&gt;
Once the execution of a job has been requested, the job is not run immediately: it is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039; by SLURM, together with all the other jobs awaiting execution. The job on top of the queue at any time is the first to be put into execution as soon as the resources it requires are available. &lt;br /&gt;
&lt;br /&gt;
The order of the items in the job queue depends on their &amp;#039;&amp;#039;&amp;#039;priority&amp;#039;&amp;#039;&amp;#039;. The job with the highest priority is put by SLURM on top of the queue, while all other jobs are put in the queue in order of descending priority.&lt;br /&gt;
&lt;br /&gt;
The goal of SLURM is to maximise resource availability: i.e., to ensure the shorter possible wait time before job execution. To achieve this goal, in Mufasa SLURM is configured to &amp;#039;&amp;#039;&amp;#039;encourage users not to ask for resources or execution time that their job doesn&amp;#039;t need&amp;#039;&amp;#039;&amp;#039;. This is done via the priority mechanism: the more resources and/or time a job requests, the lower its priority will be; and the later it will be executed.&lt;br /&gt;
&lt;br /&gt;
Priority management in Mufasa is designed to set up a &amp;#039;&amp;#039;&amp;#039;virtuous cycle&amp;#039;&amp;#039;&amp;#039; where users, by carefully choosing what to ask for, obtain two results:&lt;br /&gt;
* they ensure that their job is executed as soon as possible;&lt;br /&gt;
* they leave as much as possible of Mufasa&amp;#039;s resources free for other users&amp;#039;s jobs.&lt;br /&gt;
&lt;br /&gt;
== Elements determining job priority ==&lt;br /&gt;
In Mufasa, the priority of a job is computed by SLURM according to the following elements:&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#research users and students users|User category]]&amp;#039;&amp;#039;&amp;#039; (i.e., &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;)&lt;br /&gt;
::: Used to provide higher priority to jobs run by &amp;#039;&amp;#039;&amp;#039;research personnel&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#SLURM Quality of Service (QOS)|QOS]]&amp;#039;&amp;#039;&amp;#039; used by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;less resources&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Number of CPUs&amp;#039;&amp;#039;&amp;#039; requested by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;a lower number of CPUs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job Age&amp;#039;&amp;#039;&amp;#039;, i.e. the time that the job has been waiting in the queue&lt;br /&gt;
::: Used to provide higher priority to jobs which have been &amp;#039;&amp;#039;&amp;#039;queued for a longer time&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;FairShare&amp;#039;&amp;#039;&amp;#039;, i.e. a factor computed by SLURM to balance use of the system by different users&lt;br /&gt;
::: Used to provide higher priority to jobs by users who &amp;#039;&amp;#039;&amp;#039;used Mufasa&amp;#039;s resources less than others&amp;#039;&amp;#039;&amp;#039; (see [[#How FairShare works|below]])&lt;br /&gt;
&lt;br /&gt;
You can understand how each of these elements contribute to the priority of your jobs using command &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;. A tailored version of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; that only displays the parameters listed above, in the same order, is the following:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sprio -o &amp;quot;%6i %9u %.5Y %.10B %.5Q %.9T %.5A %.5F&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
; Important note!&lt;br /&gt;
:: The output of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; is only relevant for jobs &amp;#039;&amp;#039;&amp;#039;not yet in execution&amp;#039;&amp;#039;&amp;#039;. For an already running job, the value shown by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; is the priority that the job would get if still queued, which usually does not correspond to its actual priority.&lt;br /&gt;
&lt;br /&gt;
Here is an example of its output:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
JOBID  USER      PRIOR      ASSOC   QOS      TRES   AGE FAIRS&lt;br /&gt;
5774   itombini   3492       2500   750  cpu=-800   906   136&lt;br /&gt;
5780   fbottazzi  2266        625   750  cpu=-600   420  1071&lt;br /&gt;
5783   mzoccola   1766        625   750  cpu=-800   217   974&lt;br /&gt;
5790   amoglia    2717       2500   375  cpu=-800   156   487&lt;br /&gt;
5791   qliu       2232       2500   375  cpu=-800   138    19&lt;br /&gt;
5792   fpastori   2197       2500   375  cpu=-800    84    39&lt;br /&gt;
5793   bagostini  2034        625  1500 cpu=-1200    18  1091&lt;br /&gt;
5794   lpassoni   2517       2500   750  cpu=-800     9    58&lt;br /&gt;
5795   lpassoni   2517       2500   750  cpu=-800     9    58&lt;br /&gt;
5796   adurat     3101       2500   750  cpu=-800     8   643&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
In it, column ASSOC (i.e., association) corresponds to user category, while column TRES corresponds to the number of CPUs. Note that the contribution of CPUs to priority is &amp;#039;&amp;#039;negative&amp;#039;&amp;#039;, i.e. it is a penalty. Column PRIOR provides the overall priority of each job.&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
=== How FairShare works ===&lt;br /&gt;
In Mufasa 2.0, the FairShare of each user is computed according to these rules:&lt;br /&gt;
# It considers the quantity used by (current and past) user jobs of:&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;CPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of CPUs used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;RAM&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the amount of RAM used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;GPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of GPUs used&lt;br /&gt;
#** note: 40 GB GPUs have double the impact on FairShare than 20 GB GPUs&lt;br /&gt;
# The impact on FairShare of any use of resources is &amp;#039;&amp;#039;&amp;#039;proportional to its duration&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
#* example: using 32 GB of RAM for 48h has the same impact of using 64 GB of RAM for 24h&lt;br /&gt;
# FairShare has a &amp;quot;fading memory&amp;quot;: i.e., resource use has less and less impact on FairShare the farther in the past it is&lt;br /&gt;
#* consequence: over time, the &amp;quot;history&amp;quot; of a user gets forgotten by FairShare&lt;br /&gt;
&lt;br /&gt;
== How to maximise the priority of your jobs ==&lt;br /&gt;
&lt;br /&gt;
Every time you run a SLURM job, follow these guidelines:&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Choose the less powerful QOS compatible with the needs of your job&lt;br /&gt;
:: QOSes with access to less resources have higher base priority&lt;br /&gt;
&lt;br /&gt;
; Only request CPUs and RAM that your job really needs&lt;br /&gt;
:: Usually, code exploits multiple CPUs only if specifically designed to do so: if unsure, only ask for 1 CPU&lt;br /&gt;
:: The fewer resources your job asks for, the less it will wait before they become available&lt;br /&gt;
:: Asking for fewer CPUs and/or less RAM improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Do not request more time than your jobs needs to complete&lt;br /&gt;
:: Make a worst-case estimate and only ask for that duration&lt;br /&gt;
:: Asking for less of Mufasa&amp;#039;s time improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Debug and test code using less powerful QOSes before running it with more powerful QOSes&lt;br /&gt;
:: Your test jobs will get a higher priority and your FairShare will improve&lt;br /&gt;
&lt;br /&gt;
; Cancel jobs that you don&amp;#039;t need them anymore&lt;br /&gt;
:: Always use [[User_Jobs#Cancelling_a_job_with_scancel|scancel]] to delete completed (or crashed) jobs: your Fairshare will improve&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Protip: it&amp;#039;s a good idea to [[User Jobs#Looking for unused GPUs|check for unused GPUs]] before choosing what to request. Requesting a GPU that is currently idle will help your job get executed sooner.&lt;br /&gt;
&lt;br /&gt;
= SLURM partitions =&lt;br /&gt;
&lt;br /&gt;
Partitions are another mechanism provided by SLURM to create different levels of access to system resources. Since in Mufasa 2.0 access to resources is controlled via [[#SLURM Quality of Service (QOS)|QOSes]], partitions are not very relevant. (This is one of the main differences between Mufasa 1.0 and Mufasa 2.0.)&lt;br /&gt;
&lt;br /&gt;
Note, however, that the default values for some features of SLURM jobs (e.g., duration) are [[#Default values|set by the partition]].&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, there is a single SLURM partition, called &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt;, and all jobs run on it. The state of &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt; can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sinfo -o &amp;quot;%10P %5a %9T %11L %10l&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PARTITION  AVAIL STATE     DEFAULTTIME TIMELIMIT &lt;br /&gt;
jobs*      up    idle      1:00:00     3-00:00:00&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where columns correspond to the following information:&lt;br /&gt;
&lt;br /&gt;
:; PARTITION&lt;br /&gt;
:: name of the partition; the asterisks indicates that it&amp;#039;s the default one&lt;br /&gt;
&lt;br /&gt;
:; AVAIL&lt;br /&gt;
:: state/availability of the partition: see [[#Partition availability|below]]&lt;br /&gt;
&lt;br /&gt;
:; STATE&lt;br /&gt;
:: state (using [https://slurm.schedmd.com/sinfo.html#SECTION_NODE-STATE-CODES these codes])&lt;br /&gt;
:: typical values are &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;mixed&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that some of the resources are busy executing jobs while other are idle, and &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;allocated&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that all of the resources are in use&lt;br /&gt;
&lt;br /&gt;
:; DEFAULTTIME&lt;br /&gt;
:: default runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:; TIMELIMIT&lt;br /&gt;
:: maximum runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
The asterisk at the end of the partition name indicates the default partition, i.e. the one on which jobs which do not ask for a specific partition are run.&lt;br /&gt;
&lt;br /&gt;
Command &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; does not tell you about the &amp;#039;&amp;#039;jobs&amp;#039;&amp;#039; submitted to a partition. This information is obtained, instead, with [[User Jobs#Inspecting jobs with squeue|command &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;]].&lt;br /&gt;
&lt;br /&gt;
== Partition availability ==&lt;br /&gt;
&lt;br /&gt;
The most important information that &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; provides is the &amp;#039;&amp;#039;&amp;#039;availability&amp;#039;&amp;#039;&amp;#039; (also called &amp;#039;&amp;#039;&amp;#039;state&amp;#039;&amp;#039;&amp;#039;) of partitions. This is shown in column &amp;quot;AVAIL&amp;quot;. Possible partition states are:&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is available&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed as soon as resources allow&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is in the process of becoming unavailable (i.e., of entering the &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; state: see below)&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is unavailable&lt;br /&gt;
:: There are no running jobs&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
When a partition goes from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; no harm is done to running jobs. In a normally functioning SLURM system, the passage from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; happens only when no jobs are running on the partition. If (e.g., due to a malfunction) the passage happens with jobs still running, they get killed.&lt;br /&gt;
&lt;br /&gt;
A partition in state &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; requires intervention by a [[Roles|Job Administrator]] to be restored to &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
== Default values ==&lt;br /&gt;
&lt;br /&gt;
The features of SLURM partitions, including the &amp;#039;&amp;#039;&amp;#039;default values&amp;#039;&amp;#039;&amp;#039; which are applied to jobs that do not make explicit requests, can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scontrol show partition&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PartitionName=jobs&lt;br /&gt;
   AllowGroups=ALL AllowAccounts=ALL AllowQos=nogpu,gpulight,gpu,gpuwide,gpuheavy-20,gpuheavy-40&lt;br /&gt;
   AllocNodes=ALL Default=YES QoS=N/A&lt;br /&gt;
=&amp;gt; DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO&lt;br /&gt;
   MaxNodes=UNLIMITED MaxTime=3-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED&lt;br /&gt;
   Nodes=gn01&lt;br /&gt;
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO&lt;br /&gt;
   OverTimeLimit=NONE PreemptMode=OFF&lt;br /&gt;
   State=UP TotalCPUs=48 TotalNodes=1 SelectTypeParameters=NONE&lt;br /&gt;
   JobDefaults=(null)&lt;br /&gt;
=&amp;gt; DefMemPerNode=4096 MaxMemPerNode=UNLIMITED&lt;br /&gt;
   TRES=cpu=48,mem=1011435M,node=1,billing=49,gres/gpu=13,gres/gpu:3g.20gb=5,gres/gpu:40gb=3,gres/gpu:4g.20gb=5&lt;br /&gt;
   TRESBillingWeights=cpu=1.0,gres/gpu:3g.20gb=6.0,gres/gpu:4g.20gb=6.0,gres/gpu:40gb=6.0,mem=0.05g&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In the example, we have highlighted with  &amp;#039;&amp;#039;&amp;#039;=&amp;gt;&amp;#039;&amp;#039;&amp;#039;  the most relevant default values for Mufasa users, i.e.:&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefaultTime&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default execution time assigned to a job run on the partition (e.g., 1 hour)&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefMemPerNode&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default amount of RAM assigned to a job run on the partition (e.g., 4GB)&lt;br /&gt;
&lt;br /&gt;
= System resources subjected to limitations =&lt;br /&gt;
&lt;br /&gt;
In systems based on SLURM like Mufasa, &amp;#039;&amp;#039;&amp;#039;TRES (Trackable RESources)&amp;#039;&amp;#039;&amp;#039; are (from [https://slurm.schedmd.com/tres.html SLURM&amp;#039;s documentation] &amp;quot;&amp;#039;&amp;#039;resources that can be tracked for usage or used to enforce limits against.&amp;#039;&amp;#039;&amp;quot;&lt;br /&gt;
&lt;br /&gt;
TRES include CPUs, RAM and &amp;#039;&amp;#039;&amp;#039;GRES&amp;#039;&amp;#039;&amp;#039;. The last term stands for &amp;#039;&amp;#039;Generic RESources&amp;#039;&amp;#039; that a job may need for its execution. In Mufasa, the only &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; resources are the GPUs.&lt;br /&gt;
&lt;br /&gt;
== &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax ==&lt;br /&gt;
&lt;br /&gt;
To ask SLURM to assign GRES resources (i.e., GPUs) to a job, a special syntax must be used. Precisely, the name of each GPU resource takes the form&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:Name:Type&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
Considering the [[System#CPUs and GPUs|GPU complement of Mufasa]], &amp;lt;code&amp;gt;Type&amp;lt;/code&amp;gt; takes the following values:&lt;br /&gt;
&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:40gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 40 Gbytes of RAM&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:4g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 4 compute units&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 3 compute units&lt;br /&gt;
&lt;br /&gt;
So, for instance,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
identifies a resource corresponding to a GPU with 20 GB of RAM and 3 compute units.&lt;br /&gt;
&lt;br /&gt;
When asking for a GRES resource (e.g., in an &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command or an &amp;lt;code&amp;gt;SBATCH&amp;lt;/code&amp;gt; directive of an [[User Jobs#Using execution scripts to run jobs|execution script]]), the syntax required by SLURM is&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:&amp;lt;Type&amp;gt;:&amp;lt;Quantity&amp;gt;&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
where &amp;lt;code&amp;gt;Quantity&amp;lt;/code&amp;gt; is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of type &amp;lt;code&amp;gt;4g.20gb&amp;lt;/code&amp;gt; the syntax is&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:4g.20gb:2&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SLURM&amp;#039;s &amp;#039;&amp;#039;generic resources&amp;#039;&amp;#039; are defined in &amp;lt;code&amp;gt;/etc/slurm/gres.conf&amp;lt;/code&amp;gt;. In order to make GPUs available to SLURM&amp;#039;s &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; management, Mufasa makes use of Nvidia&amp;#039;s [https://developer.nvidia.com/nvidia-management-library-nvml NVML library]. For additional information see [https://slurm.schedmd.com/gres.html SLURM&amp;#039;s documentation].&lt;/div&gt;</summary>
		<author><name>GiulioFontana</name></author>
	</entry>
	<entry>
		<id>https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2571</id>
		<title>SLURM</title>
		<link rel="alternate" type="text/html" href="https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2571"/>
		<updated>2026-11-03T14:36:18Z</updated>

		<summary type="html">&lt;p&gt;GiulioFontana: /* Elements determining job priority */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page presents the features of SLURM that are most relevant to Mufasa&amp;#039;s [[Roles|Job Users]]. Job Users can submit jobs for execution, cancel their own jobs, and see other users&amp;#039; jobs (but not intervene on them).&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa &amp;#039;&amp;#039;&amp;#039;must use SLURM&amp;#039;&amp;#039;&amp;#039; to run resource-heavy processes, i.e. computing jobs that require one or more of the following:&lt;br /&gt;
* GPUs&lt;br /&gt;
* multiple CPUs&lt;br /&gt;
* powerful CPUs&lt;br /&gt;
* a significant amount of RAM&lt;br /&gt;
&lt;br /&gt;
In fact, only processes run via SLURM have access to all the resources of Mufasa. Processes run outside SLURM are executed by the [[System#Login server|login server]] virtual machine, which has minimal resources and no GPUs. Using SLURM is therefore the only way to execute resource-heavy jobs on Mufasa (this is a key difference between Mufasa 1.0 and Mufasa 2.0).&lt;br /&gt;
&lt;br /&gt;
= SLURM in a nutshell =&lt;br /&gt;
&lt;br /&gt;
Computation jobs on Mufasa needs to be launched via [[System#The SLURM job scheduling system|SLURM]]. SLURM provides jobs with access to the [[#System resources subjected to limitations|physical resources]] of Mufasa, such as CPUs, GPUs and RAM. Thanks to SLURM, processing jobs share system resources, optimising their occupation and availability. &lt;br /&gt;
&lt;br /&gt;
When a user runs a job, the job does not get executed immediately and is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039;. SLURM executes jobs according to their order in the queue: the top job in the queue gets executed as soon as the necessary resources are available, while jobs lower in the queue wait longer. The position of a job in the queue is due to the &amp;#039;&amp;#039;&amp;#039;[[#Job priority|priority]]&amp;#039;&amp;#039;&amp;#039; assigned to it by SLURM, with higher-priority jobs closer to the top. As a general rule:&lt;br /&gt;
&lt;br /&gt;
;: &amp;#039;&amp;#039;&amp;#039;the greater the fraction of Mufasa&amp;#039;s overall resources that a job asks for, the lower the job&amp;#039;s priority will be&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
The priority mechanism is used to encourage users to use Mufasa&amp;#039;s resources (i.e.: GPUs, CPUs, RAM, execution time) in an effective and equitable manner. This page includes a [[#How_to_maximise_the_priority_of_your_jobs|chart explaining how to maximise the priority of your jobs]].&lt;br /&gt;
&lt;br /&gt;
The &amp;#039;&amp;#039;&amp;#039;time&amp;#039;&amp;#039;&amp;#039; available to a job for its execution is controlled by SLURM. When a user requests execution of a job, they must specify the duration of the time slot that the job needs. The job must complete its execution before the end of the requested time slot, because at the end of such slot it gets killed by SLURM.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0 access to system resources is managed via SLURM&amp;#039;s [[#SLURM Quality of Service (QOS)|Quality of Service (QOS)]] mechanism (Mufasa 1.0 used [[#SLURM_partitions|partitions]] instead). While [[User Jobs#Running jobs with SLURM|launching a processing job via SLURM]], the user must always specify the chosen QOS. QOSes differ in the set of resources that they provide access to because each of them is designed to fit a given type of job.&lt;br /&gt;
&lt;br /&gt;
Mufasa sets limits to the number of jobs by the same user. This page includes a [[#Limits on jobs by the same user|table summarising such limits]].&lt;br /&gt;
&lt;br /&gt;
= SLURM Quality of Service (QOS) =&lt;br /&gt;
&lt;br /&gt;
Through &amp;#039;&amp;#039;&amp;#039;Quality of Services&amp;#039;&amp;#039;&amp;#039; (&amp;#039;&amp;#039;&amp;#039;QOSes&amp;#039;&amp;#039;&amp;#039;), SLURM lets system configurators assign a name to a set of related constraints.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, QOSes are used to define different levels of access to the server&amp;#039;s resources. When [[User Jobs|executing a job with SLURM]], a user must always &amp;#039;&amp;#039;&amp;#039;specify the QOS&amp;#039;&amp;#039;&amp;#039; that their job will use: this choice, in turn, determines what resources the job is able to access and influences the [[#Job priority|priority]] of the job.&lt;br /&gt;
&lt;br /&gt;
Mufasa&amp;#039;s QOSes and their features can be inspected with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list qos format=name%-11,priority,MaxSubmit,maxwall,maxtres%-80,mintres%-18&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following (see below for the same data in table form):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
Name          Priority MaxSubmit     MaxWall MaxTRES                                                                          MinTRES            &lt;br /&gt;
----------- ---------- --------- ----------- -------------------------------------------------------------------------------- ------------------ &lt;br /&gt;
normal               0                                                                                                                           &lt;br /&gt;
nogpu                4         1  3-00:00:00 cpu=16,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=128G                    &lt;br /&gt;
gpuheavy-20          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=2,mem=128G             gres/gpu:4g.20gb=1 &lt;br /&gt;
gpuheavy-40          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=1,gres/gpu:4g.20gb=0,mem=128G             gres/gpu:40gb=1    &lt;br /&gt;
gpulight             8         1    12:00:00 cpu=2,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpu                  2         1  1-00:00:00 cpu=8,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpuwide              2         2  1-00:00:00 cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=1,mem=64G              gres/gpu:4g.20gb=1 &lt;br /&gt;
build               32         1    02:00:00 cpu=2,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=16G&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The columns of this output are the following:&lt;br /&gt;
&lt;br /&gt;
:; Name&lt;br /&gt;
:: name of the QOS&lt;br /&gt;
&lt;br /&gt;
:; Priority&lt;br /&gt;
:: priority tier associated to the QOS (higher value = higher priority): see [[#Job priority|Job priority]] for details&lt;br /&gt;
&lt;br /&gt;
:; MaxSubmit&lt;br /&gt;
:: maximum number of jobs from a single user that can be submitted to SLURM with this QOS; submitted jobs include both running and queued jobs&lt;br /&gt;
:: See [[#Limits on jobs by the same user|Limits on jobs by the same user]] for an overview of the limits on jobs set by Mufasa.&lt;br /&gt;
&lt;br /&gt;
:; MaxWall&lt;br /&gt;
:: maximum wall clock duration of the jobs using the QOS (after which they are killed by SLURM), in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
:: For some QOSes these are not set: it means that they are determined by the [[#SLURM partitions|partition]]. Partitions also define the [[#Default values|default duration]] of jobs.&lt;br /&gt;
&lt;br /&gt;
:; MaxTRES&lt;br /&gt;
:: amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) available to a job using the QOS, where&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;cpu=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of CPUs (i.e., processor cores) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of CPUs specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;gres/&amp;#039;&amp;#039;gpu:Type&amp;#039;&amp;#039;=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of GPUs of class &amp;lt;code&amp;gt;&amp;#039;&amp;#039;Type&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; (see [[User Jobs#gres syntax|&amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax]]) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; (for QOSes that allow access to GPUs) if not specified, the job cannot be launched&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;mem=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;G&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum amount of system RAM is &amp;#039;&amp;#039;K&amp;#039;&amp;#039; GBytes&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of RAM specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
&lt;br /&gt;
:; MinTRES&lt;br /&gt;
:: minimum amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) that a job using the QOS must request in order to actually get executed by SLURM.&lt;br /&gt;
:: (If your job does not actually need these resources, you&amp;#039;ve chosen the wrong QOS: you can use one with a higher priority.)&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; QOS is the default one, and exists only to ensure that users always specify a QOS when running a job. Since &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; has zero priority and no resources, a job run using this QOS would never be run.&lt;br /&gt;
&lt;br /&gt;
The information provided by the &amp;lt;code&amp;gt;sacctmgr list qos&amp;lt;/code&amp;gt; command above is summarised by the following table:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! QOS&lt;br /&gt;
! Priority&amp;lt;/br&amp;gt;tier&lt;br /&gt;
! Max&amp;lt;/br&amp;gt;Submit&lt;br /&gt;
! MaxWall&amp;lt;/br&amp;gt;[h]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;CPUs&lt;br /&gt;
! max RAM&amp;lt;/br&amp;gt;[GB]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;3g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;4g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;40GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | build&lt;br /&gt;
| 32&lt;br /&gt;
| 1&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 16&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpulight&lt;br /&gt;
| 8&lt;br /&gt;
| 1&lt;br /&gt;
| 12&lt;br /&gt;
| 2&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | nogpu&lt;br /&gt;
| 4&lt;br /&gt;
| 1&lt;br /&gt;
| 72&lt;br /&gt;
| 16&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpu&lt;br /&gt;
| 2&lt;br /&gt;
| 1&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuwide&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-20&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| 2 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-40&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
A key piece of information in the table above is the &amp;#039;&amp;#039;&amp;#039;priority tier&amp;#039;&amp;#039;&amp;#039; associated to each QOS. In Mufasa 2.0, priority tiers are used to encourage users to use the &amp;#039;&amp;#039;&amp;#039;least powerful QOS that is compatible with their needs&amp;#039;&amp;#039;&amp;#039;, where &amp;quot;powerful&amp;quot; means &amp;quot;rich with resources&amp;quot;. Less powerful QOSes increase the priority of the jobs that use them, so these jobs tend to be executed sooner. See [[#Job priority|Job priority]] for details about how priority affects the execution order of jobs in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
== The &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS ==&lt;br /&gt;
&lt;br /&gt;
This QOS is specifically designed to be used by Mufasa users to &amp;#039;&amp;#039;&amp;#039;quickly build [[System#Containers|container images]]&amp;#039;&amp;#039;&amp;#039;. Its associated priority tier is very high: this, combined with the fact that jobs using this QOS require few resources, means that such jobs usually get executed very soon.&lt;br /&gt;
&lt;br /&gt;
On the other hand, the limited resources and the lack of access to the GPUs make the &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS unsuitable for other tasks.&lt;br /&gt;
&lt;br /&gt;
See [[Singularity#Building Singularity images|Building Singularity images]] for directions about building Singularity container images.&lt;br /&gt;
&lt;br /&gt;
== Restricted QOSes ==&lt;br /&gt;
&lt;br /&gt;
In Mufasa, the most powerful QOSes are reserved to researchers (called &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users: these include academic personnel and Ph.D. students). M.Sc. students, called &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users, cannot use them while running jobs.&lt;br /&gt;
&lt;br /&gt;
See [[#research users and students users|below]] to understand how &amp;#039;&amp;#039;user categories&amp;#039;&amp;#039; work in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
= &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users =&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa belong to two &amp;#039;&amp;#039;&amp;#039;user categories&amp;#039;&amp;#039;&amp;#039;. Categories provide users with different access to  to Mufasa&amp;#039;s resources: the idea being to provide researchers with more access while still letting students benefit from the server&amp;#039;s capabilities.&lt;br /&gt;
&lt;br /&gt;
User categories in Mufasa are the following:&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. academic personnel and Ph.D. students&lt;br /&gt;
::: * can use all QOSes, including the the [[#Restricted QOSes|restricted ones]]&lt;br /&gt;
::: * their jobs have a higher &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a higher number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. M.Sc. students&lt;br /&gt;
::: * cannot use the [[#Restricted QOSes|restricted QOSes]]&lt;br /&gt;
::: * their jobs have a lower &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a lower number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
You can inspect the differences in priority and running jobs between &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association format=&amp;quot;account,priority,maxjobs&amp;quot; | grep -E &amp;#039;Account|research|students&amp;#039;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
   Account   Priority MaxJobs &lt;br /&gt;
  research          4       2 &lt;br /&gt;
  students          1       1&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
To know what limits apply to your own user, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association where user=$USER format=&amp;quot;user,priority,maxjobs,qos%-60&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
      User   Priority MaxJobs QOS                                                          &lt;br /&gt;
---------- ---------- ------- ------------------------------------------------------------ &lt;br /&gt;
    preali          4       2 build,gpu,gpuheavy-20,gpuheavy-40,gpulight,gpuwide,nogpu&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The list under &amp;quot;QOS&amp;quot; shows what QOSes your user is allowed to use when [[User Jobs|running jobs]]. &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users can use all of them, while &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users can only access a subset of them.&lt;br /&gt;
&lt;br /&gt;
= Limits on jobs by the same user =&lt;br /&gt;
&lt;br /&gt;
Mufasa enforces limits on the number of jobs from a single user. Such limits aim at preventing users from &amp;quot;hogging&amp;quot; system resources, and apply to:&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;submitted jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that the user asked SLURM to execute, each of which may currently be either running or queued&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;running jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that are currently in execution&lt;br /&gt;
&lt;br /&gt;
Limits on submitted jobs are set via [[#SLURM Quality of Service (QOS)|QOSes]], while limits on running jobs are set via [[#research users and students users|user category]].&lt;br /&gt;
&lt;br /&gt;
The following table summarises what limits exist:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! &lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;running&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;submitted&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | global limits&amp;lt;br/&amp;gt;(system-wide)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to limits set by individual QOSes (below)&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | limits for&amp;lt;br/&amp;gt;individual&amp;lt;/br&amp;gt;QOSes&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to global limits (above)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for the&amp;#039;&amp;#039; &amp;lt;code&amp;gt;gpuwide&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;QOS&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for each of the other QOSes&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
= Job priority =&lt;br /&gt;
&lt;br /&gt;
Once the execution of a job has been requested, the job is not run immediately: it is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039; by SLURM, together with all the other jobs awaiting execution. The job on top of the queue at any time is the first to be put into execution as soon as the resources it requires are available. &lt;br /&gt;
&lt;br /&gt;
The order of the items in the job queue depends on their &amp;#039;&amp;#039;&amp;#039;priority&amp;#039;&amp;#039;&amp;#039;. The job with the highest priority is put by SLURM on top of the queue, while all other jobs are put in the queue in order of descending priority.&lt;br /&gt;
&lt;br /&gt;
The goal of SLURM is to maximise resource availability: i.e., to ensure the shorter possible wait time before job execution. To achieve this goal, in Mufasa SLURM is configured to &amp;#039;&amp;#039;&amp;#039;encourage users not to ask for resources or execution time that their job doesn&amp;#039;t need&amp;#039;&amp;#039;&amp;#039;. This is done via the priority mechanism: the more resources and/or time a job requests, the lower its priority will be; and the later it will be executed.&lt;br /&gt;
&lt;br /&gt;
Priority management in Mufasa is designed to set up a &amp;#039;&amp;#039;&amp;#039;virtuous cycle&amp;#039;&amp;#039;&amp;#039; where users, by carefully choosing what to ask for, obtain two results:&lt;br /&gt;
* they ensure that their job is executed as soon as possible;&lt;br /&gt;
* they leave as much as possible of Mufasa&amp;#039;s resources free for other users&amp;#039;s jobs.&lt;br /&gt;
&lt;br /&gt;
== Elements determining job priority ==&lt;br /&gt;
In Mufasa, the priority of a job is computed by SLURM according to the following elements:&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#research users and students users|User category]]&amp;#039;&amp;#039;&amp;#039; (i.e., &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;)&lt;br /&gt;
::: Used to provide higher priority to jobs run by &amp;#039;&amp;#039;&amp;#039;research personnel&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#SLURM Quality of Service (QOS)|QOS]]&amp;#039;&amp;#039;&amp;#039; used by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;less resources&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Number of CPUs&amp;#039;&amp;#039;&amp;#039; requested by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;a lower number of CPUs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job Age&amp;#039;&amp;#039;&amp;#039;, i.e. the time that the job has been waiting in the queue&lt;br /&gt;
::: Used to provide higher priority to jobs which have been &amp;#039;&amp;#039;&amp;#039;queued for a longer time&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;FairShare&amp;#039;&amp;#039;&amp;#039;, i.e. a factor computed by SLURM to balance use of the system by different users&lt;br /&gt;
::: Used to provide higher priority to jobs by users who &amp;#039;&amp;#039;&amp;#039;used Mufasa&amp;#039;s resources less than others&amp;#039;&amp;#039;&amp;#039; (see [[#How FairShare works|below]])&lt;br /&gt;
&lt;br /&gt;
You can understand how each of these elements contribute to the priority of your jobs using command &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;. A tailored version of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; that only displays the parameters listed above, in the same order, is the following:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sprio -o &amp;quot;%6i %9u %.5Y %.10B %.5Q %.9T %.5A %.5F&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
Here is an example of its output:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
JOBID  USER      PRIOR      ASSOC   QOS      TRES   AGE FAIRS&lt;br /&gt;
5774   itombini   3492       2500   750  cpu=-800   906   136&lt;br /&gt;
5780   fbottazzi  2266        625   750  cpu=-600   420  1071&lt;br /&gt;
5783   mzoccola   1766        625   750  cpu=-800   217   974&lt;br /&gt;
5790   amoglia    2717       2500   375  cpu=-800   156   487&lt;br /&gt;
5791   qliu       2232       2500   375  cpu=-800   138    19&lt;br /&gt;
5792   fpastori   2197       2500   375  cpu=-800    84    39&lt;br /&gt;
5793   bagostini  2034        625  1500 cpu=-1200    18  1091&lt;br /&gt;
5794   lpassoni   2517       2500   750  cpu=-800     9    58&lt;br /&gt;
5795   lpassoni   2517       2500   750  cpu=-800     9    58&lt;br /&gt;
5796   adurat     3101       2500   750  cpu=-800     8   643&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
In it, column ASSOC (i.e., association) corresponds to user category, while column TRES corresponds to the number of CPUs. Note that the contribution of CPUs to priority is &amp;#039;&amp;#039;negative&amp;#039;&amp;#039;, i.e. it is a penalty. Column PRIOR provides the overall priority of each job.&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Important note&lt;br /&gt;
:: The output of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; is only relevant for jobs &amp;#039;&amp;#039;&amp;#039;not yet in execution&amp;#039;&amp;#039;&amp;#039;. For an already running job, the value shown by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; is the priority that the job would get if still queued, which usually does not correspond to its actual priority.&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
=== How FairShare works ===&lt;br /&gt;
In Mufasa 2.0, the FairShare of each user is computed according to these rules:&lt;br /&gt;
# It considers the quantity used by (current and past) user jobs of:&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;CPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of CPUs used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;RAM&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the amount of RAM used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;GPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of GPUs used&lt;br /&gt;
#** note: 40 GB GPUs have double the impact on FairShare than 20 GB GPUs&lt;br /&gt;
# The impact on FairShare of any use of resources is &amp;#039;&amp;#039;&amp;#039;proportional to its duration&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
#* example: using 32 GB of RAM for 48h has the same impact of using 64 GB of RAM for 24h&lt;br /&gt;
# FairShare has a &amp;quot;fading memory&amp;quot;: i.e., resource use has less and less impact on FairShare the farther in the past it is&lt;br /&gt;
#* consequence: over time, the &amp;quot;history&amp;quot; of a user gets forgotten by FairShare&lt;br /&gt;
&lt;br /&gt;
== How to maximise the priority of your jobs ==&lt;br /&gt;
&lt;br /&gt;
Every time you run a SLURM job, follow these guidelines:&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Choose the less powerful QOS compatible with the needs of your job&lt;br /&gt;
:: QOSes with access to less resources have higher base priority&lt;br /&gt;
&lt;br /&gt;
; Only request CPUs and RAM that your job really needs&lt;br /&gt;
:: Usually, code exploits multiple CPUs only if specifically designed to do so: if unsure, only ask for 1 CPU&lt;br /&gt;
:: The fewer resources your job asks for, the less it will wait before they become available&lt;br /&gt;
:: Asking for fewer CPUs and/or less RAM improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Do not request more time than your jobs needs to complete&lt;br /&gt;
:: Make a worst-case estimate and only ask for that duration&lt;br /&gt;
:: Asking for less of Mufasa&amp;#039;s time improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Debug and test code using less powerful QOSes before running it with more powerful QOSes&lt;br /&gt;
:: Your test jobs will get a higher priority and your FairShare will improve&lt;br /&gt;
&lt;br /&gt;
; Cancel jobs that you don&amp;#039;t need them anymore&lt;br /&gt;
:: Always use [[User_Jobs#Cancelling_a_job_with_scancel|scancel]] to delete completed (or crashed) jobs: your Fairshare will improve&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Protip: it&amp;#039;s a good idea to [[User Jobs#Looking for unused GPUs|check for unused GPUs]] before choosing what to request. Requesting a GPU that is currently idle will help your job get executed sooner.&lt;br /&gt;
&lt;br /&gt;
= SLURM partitions =&lt;br /&gt;
&lt;br /&gt;
Partitions are another mechanism provided by SLURM to create different levels of access to system resources. Since in Mufasa 2.0 access to resources is controlled via [[#SLURM Quality of Service (QOS)|QOSes]], partitions are not very relevant. (This is one of the main differences between Mufasa 1.0 and Mufasa 2.0.)&lt;br /&gt;
&lt;br /&gt;
Note, however, that the default values for some features of SLURM jobs (e.g., duration) are [[#Default values|set by the partition]].&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, there is a single SLURM partition, called &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt;, and all jobs run on it. The state of &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt; can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sinfo -o &amp;quot;%10P %5a %9T %11L %10l&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PARTITION  AVAIL STATE     DEFAULTTIME TIMELIMIT &lt;br /&gt;
jobs*      up    idle      1:00:00     3-00:00:00&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where columns correspond to the following information:&lt;br /&gt;
&lt;br /&gt;
:; PARTITION&lt;br /&gt;
:: name of the partition; the asterisks indicates that it&amp;#039;s the default one&lt;br /&gt;
&lt;br /&gt;
:; AVAIL&lt;br /&gt;
:: state/availability of the partition: see [[#Partition availability|below]]&lt;br /&gt;
&lt;br /&gt;
:; STATE&lt;br /&gt;
:: state (using [https://slurm.schedmd.com/sinfo.html#SECTION_NODE-STATE-CODES these codes])&lt;br /&gt;
:: typical values are &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;mixed&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that some of the resources are busy executing jobs while other are idle, and &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;allocated&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that all of the resources are in use&lt;br /&gt;
&lt;br /&gt;
:; DEFAULTTIME&lt;br /&gt;
:: default runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:; TIMELIMIT&lt;br /&gt;
:: maximum runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
The asterisk at the end of the partition name indicates the default partition, i.e. the one on which jobs which do not ask for a specific partition are run.&lt;br /&gt;
&lt;br /&gt;
Command &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; does not tell you about the &amp;#039;&amp;#039;jobs&amp;#039;&amp;#039; submitted to a partition. This information is obtained, instead, with [[User Jobs#Inspecting jobs with squeue|command &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;]].&lt;br /&gt;
&lt;br /&gt;
== Partition availability ==&lt;br /&gt;
&lt;br /&gt;
The most important information that &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; provides is the &amp;#039;&amp;#039;&amp;#039;availability&amp;#039;&amp;#039;&amp;#039; (also called &amp;#039;&amp;#039;&amp;#039;state&amp;#039;&amp;#039;&amp;#039;) of partitions. This is shown in column &amp;quot;AVAIL&amp;quot;. Possible partition states are:&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is available&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed as soon as resources allow&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is in the process of becoming unavailable (i.e., of entering the &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; state: see below)&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is unavailable&lt;br /&gt;
:: There are no running jobs&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
When a partition goes from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; no harm is done to running jobs. In a normally functioning SLURM system, the passage from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; happens only when no jobs are running on the partition. If (e.g., due to a malfunction) the passage happens with jobs still running, they get killed.&lt;br /&gt;
&lt;br /&gt;
A partition in state &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; requires intervention by a [[Roles|Job Administrator]] to be restored to &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
== Default values ==&lt;br /&gt;
&lt;br /&gt;
The features of SLURM partitions, including the &amp;#039;&amp;#039;&amp;#039;default values&amp;#039;&amp;#039;&amp;#039; which are applied to jobs that do not make explicit requests, can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scontrol show partition&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PartitionName=jobs&lt;br /&gt;
   AllowGroups=ALL AllowAccounts=ALL AllowQos=nogpu,gpulight,gpu,gpuwide,gpuheavy-20,gpuheavy-40&lt;br /&gt;
   AllocNodes=ALL Default=YES QoS=N/A&lt;br /&gt;
=&amp;gt; DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO&lt;br /&gt;
   MaxNodes=UNLIMITED MaxTime=3-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED&lt;br /&gt;
   Nodes=gn01&lt;br /&gt;
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO&lt;br /&gt;
   OverTimeLimit=NONE PreemptMode=OFF&lt;br /&gt;
   State=UP TotalCPUs=48 TotalNodes=1 SelectTypeParameters=NONE&lt;br /&gt;
   JobDefaults=(null)&lt;br /&gt;
=&amp;gt; DefMemPerNode=4096 MaxMemPerNode=UNLIMITED&lt;br /&gt;
   TRES=cpu=48,mem=1011435M,node=1,billing=49,gres/gpu=13,gres/gpu:3g.20gb=5,gres/gpu:40gb=3,gres/gpu:4g.20gb=5&lt;br /&gt;
   TRESBillingWeights=cpu=1.0,gres/gpu:3g.20gb=6.0,gres/gpu:4g.20gb=6.0,gres/gpu:40gb=6.0,mem=0.05g&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In the example, we have highlighted with  &amp;#039;&amp;#039;&amp;#039;=&amp;gt;&amp;#039;&amp;#039;&amp;#039;  the most relevant default values for Mufasa users, i.e.:&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefaultTime&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default execution time assigned to a job run on the partition (e.g., 1 hour)&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefMemPerNode&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default amount of RAM assigned to a job run on the partition (e.g., 4GB)&lt;br /&gt;
&lt;br /&gt;
= System resources subjected to limitations =&lt;br /&gt;
&lt;br /&gt;
In systems based on SLURM like Mufasa, &amp;#039;&amp;#039;&amp;#039;TRES (Trackable RESources)&amp;#039;&amp;#039;&amp;#039; are (from [https://slurm.schedmd.com/tres.html SLURM&amp;#039;s documentation] &amp;quot;&amp;#039;&amp;#039;resources that can be tracked for usage or used to enforce limits against.&amp;#039;&amp;#039;&amp;quot;&lt;br /&gt;
&lt;br /&gt;
TRES include CPUs, RAM and &amp;#039;&amp;#039;&amp;#039;GRES&amp;#039;&amp;#039;&amp;#039;. The last term stands for &amp;#039;&amp;#039;Generic RESources&amp;#039;&amp;#039; that a job may need for its execution. In Mufasa, the only &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; resources are the GPUs.&lt;br /&gt;
&lt;br /&gt;
== &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax ==&lt;br /&gt;
&lt;br /&gt;
To ask SLURM to assign GRES resources (i.e., GPUs) to a job, a special syntax must be used. Precisely, the name of each GPU resource takes the form&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:Name:Type&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
Considering the [[System#CPUs and GPUs|GPU complement of Mufasa]], &amp;lt;code&amp;gt;Type&amp;lt;/code&amp;gt; takes the following values:&lt;br /&gt;
&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:40gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 40 Gbytes of RAM&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:4g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 4 compute units&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 3 compute units&lt;br /&gt;
&lt;br /&gt;
So, for instance,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
identifies a resource corresponding to a GPU with 20 GB of RAM and 3 compute units.&lt;br /&gt;
&lt;br /&gt;
When asking for a GRES resource (e.g., in an &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command or an &amp;lt;code&amp;gt;SBATCH&amp;lt;/code&amp;gt; directive of an [[User Jobs#Using execution scripts to run jobs|execution script]]), the syntax required by SLURM is&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:&amp;lt;Type&amp;gt;:&amp;lt;Quantity&amp;gt;&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
where &amp;lt;code&amp;gt;Quantity&amp;lt;/code&amp;gt; is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of type &amp;lt;code&amp;gt;4g.20gb&amp;lt;/code&amp;gt; the syntax is&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:4g.20gb:2&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SLURM&amp;#039;s &amp;#039;&amp;#039;generic resources&amp;#039;&amp;#039; are defined in &amp;lt;code&amp;gt;/etc/slurm/gres.conf&amp;lt;/code&amp;gt;. In order to make GPUs available to SLURM&amp;#039;s &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; management, Mufasa makes use of Nvidia&amp;#039;s [https://developer.nvidia.com/nvidia-management-library-nvml NVML library]. For additional information see [https://slurm.schedmd.com/gres.html SLURM&amp;#039;s documentation].&lt;/div&gt;</summary>
		<author><name>GiulioFontana</name></author>
	</entry>
	<entry>
		<id>https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2570</id>
		<title>SLURM</title>
		<link rel="alternate" type="text/html" href="https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2570"/>
		<updated>2026-11-03T14:34:32Z</updated>

		<summary type="html">&lt;p&gt;GiulioFontana: /* Elements determining job priority */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page presents the features of SLURM that are most relevant to Mufasa&amp;#039;s [[Roles|Job Users]]. Job Users can submit jobs for execution, cancel their own jobs, and see other users&amp;#039; jobs (but not intervene on them).&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa &amp;#039;&amp;#039;&amp;#039;must use SLURM&amp;#039;&amp;#039;&amp;#039; to run resource-heavy processes, i.e. computing jobs that require one or more of the following:&lt;br /&gt;
* GPUs&lt;br /&gt;
* multiple CPUs&lt;br /&gt;
* powerful CPUs&lt;br /&gt;
* a significant amount of RAM&lt;br /&gt;
&lt;br /&gt;
In fact, only processes run via SLURM have access to all the resources of Mufasa. Processes run outside SLURM are executed by the [[System#Login server|login server]] virtual machine, which has minimal resources and no GPUs. Using SLURM is therefore the only way to execute resource-heavy jobs on Mufasa (this is a key difference between Mufasa 1.0 and Mufasa 2.0).&lt;br /&gt;
&lt;br /&gt;
= SLURM in a nutshell =&lt;br /&gt;
&lt;br /&gt;
Computation jobs on Mufasa needs to be launched via [[System#The SLURM job scheduling system|SLURM]]. SLURM provides jobs with access to the [[#System resources subjected to limitations|physical resources]] of Mufasa, such as CPUs, GPUs and RAM. Thanks to SLURM, processing jobs share system resources, optimising their occupation and availability. &lt;br /&gt;
&lt;br /&gt;
When a user runs a job, the job does not get executed immediately and is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039;. SLURM executes jobs according to their order in the queue: the top job in the queue gets executed as soon as the necessary resources are available, while jobs lower in the queue wait longer. The position of a job in the queue is due to the &amp;#039;&amp;#039;&amp;#039;[[#Job priority|priority]]&amp;#039;&amp;#039;&amp;#039; assigned to it by SLURM, with higher-priority jobs closer to the top. As a general rule:&lt;br /&gt;
&lt;br /&gt;
;: &amp;#039;&amp;#039;&amp;#039;the greater the fraction of Mufasa&amp;#039;s overall resources that a job asks for, the lower the job&amp;#039;s priority will be&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
The priority mechanism is used to encourage users to use Mufasa&amp;#039;s resources (i.e.: GPUs, CPUs, RAM, execution time) in an effective and equitable manner. This page includes a [[#How_to_maximise_the_priority_of_your_jobs|chart explaining how to maximise the priority of your jobs]].&lt;br /&gt;
&lt;br /&gt;
The &amp;#039;&amp;#039;&amp;#039;time&amp;#039;&amp;#039;&amp;#039; available to a job for its execution is controlled by SLURM. When a user requests execution of a job, they must specify the duration of the time slot that the job needs. The job must complete its execution before the end of the requested time slot, because at the end of such slot it gets killed by SLURM.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0 access to system resources is managed via SLURM&amp;#039;s [[#SLURM Quality of Service (QOS)|Quality of Service (QOS)]] mechanism (Mufasa 1.0 used [[#SLURM_partitions|partitions]] instead). While [[User Jobs#Running jobs with SLURM|launching a processing job via SLURM]], the user must always specify the chosen QOS. QOSes differ in the set of resources that they provide access to because each of them is designed to fit a given type of job.&lt;br /&gt;
&lt;br /&gt;
Mufasa sets limits to the number of jobs by the same user. This page includes a [[#Limits on jobs by the same user|table summarising such limits]].&lt;br /&gt;
&lt;br /&gt;
= SLURM Quality of Service (QOS) =&lt;br /&gt;
&lt;br /&gt;
Through &amp;#039;&amp;#039;&amp;#039;Quality of Services&amp;#039;&amp;#039;&amp;#039; (&amp;#039;&amp;#039;&amp;#039;QOSes&amp;#039;&amp;#039;&amp;#039;), SLURM lets system configurators assign a name to a set of related constraints.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, QOSes are used to define different levels of access to the server&amp;#039;s resources. When [[User Jobs|executing a job with SLURM]], a user must always &amp;#039;&amp;#039;&amp;#039;specify the QOS&amp;#039;&amp;#039;&amp;#039; that their job will use: this choice, in turn, determines what resources the job is able to access and influences the [[#Job priority|priority]] of the job.&lt;br /&gt;
&lt;br /&gt;
Mufasa&amp;#039;s QOSes and their features can be inspected with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list qos format=name%-11,priority,MaxSubmit,maxwall,maxtres%-80,mintres%-18&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following (see below for the same data in table form):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
Name          Priority MaxSubmit     MaxWall MaxTRES                                                                          MinTRES            &lt;br /&gt;
----------- ---------- --------- ----------- -------------------------------------------------------------------------------- ------------------ &lt;br /&gt;
normal               0                                                                                                                           &lt;br /&gt;
nogpu                4         1  3-00:00:00 cpu=16,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=128G                    &lt;br /&gt;
gpuheavy-20          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=2,mem=128G             gres/gpu:4g.20gb=1 &lt;br /&gt;
gpuheavy-40          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=1,gres/gpu:4g.20gb=0,mem=128G             gres/gpu:40gb=1    &lt;br /&gt;
gpulight             8         1    12:00:00 cpu=2,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpu                  2         1  1-00:00:00 cpu=8,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpuwide              2         2  1-00:00:00 cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=1,mem=64G              gres/gpu:4g.20gb=1 &lt;br /&gt;
build               32         1    02:00:00 cpu=2,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=16G&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The columns of this output are the following:&lt;br /&gt;
&lt;br /&gt;
:; Name&lt;br /&gt;
:: name of the QOS&lt;br /&gt;
&lt;br /&gt;
:; Priority&lt;br /&gt;
:: priority tier associated to the QOS (higher value = higher priority): see [[#Job priority|Job priority]] for details&lt;br /&gt;
&lt;br /&gt;
:; MaxSubmit&lt;br /&gt;
:: maximum number of jobs from a single user that can be submitted to SLURM with this QOS; submitted jobs include both running and queued jobs&lt;br /&gt;
:: See [[#Limits on jobs by the same user|Limits on jobs by the same user]] for an overview of the limits on jobs set by Mufasa.&lt;br /&gt;
&lt;br /&gt;
:; MaxWall&lt;br /&gt;
:: maximum wall clock duration of the jobs using the QOS (after which they are killed by SLURM), in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
:: For some QOSes these are not set: it means that they are determined by the [[#SLURM partitions|partition]]. Partitions also define the [[#Default values|default duration]] of jobs.&lt;br /&gt;
&lt;br /&gt;
:; MaxTRES&lt;br /&gt;
:: amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) available to a job using the QOS, where&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;cpu=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of CPUs (i.e., processor cores) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of CPUs specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;gres/&amp;#039;&amp;#039;gpu:Type&amp;#039;&amp;#039;=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of GPUs of class &amp;lt;code&amp;gt;&amp;#039;&amp;#039;Type&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; (see [[User Jobs#gres syntax|&amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax]]) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; (for QOSes that allow access to GPUs) if not specified, the job cannot be launched&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;mem=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;G&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum amount of system RAM is &amp;#039;&amp;#039;K&amp;#039;&amp;#039; GBytes&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of RAM specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
&lt;br /&gt;
:; MinTRES&lt;br /&gt;
:: minimum amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) that a job using the QOS must request in order to actually get executed by SLURM.&lt;br /&gt;
:: (If your job does not actually need these resources, you&amp;#039;ve chosen the wrong QOS: you can use one with a higher priority.)&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; QOS is the default one, and exists only to ensure that users always specify a QOS when running a job. Since &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; has zero priority and no resources, a job run using this QOS would never be run.&lt;br /&gt;
&lt;br /&gt;
The information provided by the &amp;lt;code&amp;gt;sacctmgr list qos&amp;lt;/code&amp;gt; command above is summarised by the following table:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! QOS&lt;br /&gt;
! Priority&amp;lt;/br&amp;gt;tier&lt;br /&gt;
! Max&amp;lt;/br&amp;gt;Submit&lt;br /&gt;
! MaxWall&amp;lt;/br&amp;gt;[h]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;CPUs&lt;br /&gt;
! max RAM&amp;lt;/br&amp;gt;[GB]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;3g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;4g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;40GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | build&lt;br /&gt;
| 32&lt;br /&gt;
| 1&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 16&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpulight&lt;br /&gt;
| 8&lt;br /&gt;
| 1&lt;br /&gt;
| 12&lt;br /&gt;
| 2&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | nogpu&lt;br /&gt;
| 4&lt;br /&gt;
| 1&lt;br /&gt;
| 72&lt;br /&gt;
| 16&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpu&lt;br /&gt;
| 2&lt;br /&gt;
| 1&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuwide&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-20&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| 2 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-40&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
A key piece of information in the table above is the &amp;#039;&amp;#039;&amp;#039;priority tier&amp;#039;&amp;#039;&amp;#039; associated to each QOS. In Mufasa 2.0, priority tiers are used to encourage users to use the &amp;#039;&amp;#039;&amp;#039;least powerful QOS that is compatible with their needs&amp;#039;&amp;#039;&amp;#039;, where &amp;quot;powerful&amp;quot; means &amp;quot;rich with resources&amp;quot;. Less powerful QOSes increase the priority of the jobs that use them, so these jobs tend to be executed sooner. See [[#Job priority|Job priority]] for details about how priority affects the execution order of jobs in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
== The &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS ==&lt;br /&gt;
&lt;br /&gt;
This QOS is specifically designed to be used by Mufasa users to &amp;#039;&amp;#039;&amp;#039;quickly build [[System#Containers|container images]]&amp;#039;&amp;#039;&amp;#039;. Its associated priority tier is very high: this, combined with the fact that jobs using this QOS require few resources, means that such jobs usually get executed very soon.&lt;br /&gt;
&lt;br /&gt;
On the other hand, the limited resources and the lack of access to the GPUs make the &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS unsuitable for other tasks.&lt;br /&gt;
&lt;br /&gt;
See [[Singularity#Building Singularity images|Building Singularity images]] for directions about building Singularity container images.&lt;br /&gt;
&lt;br /&gt;
== Restricted QOSes ==&lt;br /&gt;
&lt;br /&gt;
In Mufasa, the most powerful QOSes are reserved to researchers (called &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users: these include academic personnel and Ph.D. students). M.Sc. students, called &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users, cannot use them while running jobs.&lt;br /&gt;
&lt;br /&gt;
See [[#research users and students users|below]] to understand how &amp;#039;&amp;#039;user categories&amp;#039;&amp;#039; work in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
= &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users =&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa belong to two &amp;#039;&amp;#039;&amp;#039;user categories&amp;#039;&amp;#039;&amp;#039;. Categories provide users with different access to  to Mufasa&amp;#039;s resources: the idea being to provide researchers with more access while still letting students benefit from the server&amp;#039;s capabilities.&lt;br /&gt;
&lt;br /&gt;
User categories in Mufasa are the following:&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. academic personnel and Ph.D. students&lt;br /&gt;
::: * can use all QOSes, including the the [[#Restricted QOSes|restricted ones]]&lt;br /&gt;
::: * their jobs have a higher &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a higher number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. M.Sc. students&lt;br /&gt;
::: * cannot use the [[#Restricted QOSes|restricted QOSes]]&lt;br /&gt;
::: * their jobs have a lower &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a lower number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
You can inspect the differences in priority and running jobs between &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association format=&amp;quot;account,priority,maxjobs&amp;quot; | grep -E &amp;#039;Account|research|students&amp;#039;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
   Account   Priority MaxJobs &lt;br /&gt;
  research          4       2 &lt;br /&gt;
  students          1       1&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
To know what limits apply to your own user, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association where user=$USER format=&amp;quot;user,priority,maxjobs,qos%-60&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
      User   Priority MaxJobs QOS                                                          &lt;br /&gt;
---------- ---------- ------- ------------------------------------------------------------ &lt;br /&gt;
    preali          4       2 build,gpu,gpuheavy-20,gpuheavy-40,gpulight,gpuwide,nogpu&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The list under &amp;quot;QOS&amp;quot; shows what QOSes your user is allowed to use when [[User Jobs|running jobs]]. &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users can use all of them, while &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users can only access a subset of them.&lt;br /&gt;
&lt;br /&gt;
= Limits on jobs by the same user =&lt;br /&gt;
&lt;br /&gt;
Mufasa enforces limits on the number of jobs from a single user. Such limits aim at preventing users from &amp;quot;hogging&amp;quot; system resources, and apply to:&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;submitted jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that the user asked SLURM to execute, each of which may currently be either running or queued&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;running jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that are currently in execution&lt;br /&gt;
&lt;br /&gt;
Limits on submitted jobs are set via [[#SLURM Quality of Service (QOS)|QOSes]], while limits on running jobs are set via [[#research users and students users|user category]].&lt;br /&gt;
&lt;br /&gt;
The following table summarises what limits exist:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! &lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;running&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;submitted&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | global limits&amp;lt;br/&amp;gt;(system-wide)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to limits set by individual QOSes (below)&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | limits for&amp;lt;br/&amp;gt;individual&amp;lt;/br&amp;gt;QOSes&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to global limits (above)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for the&amp;#039;&amp;#039; &amp;lt;code&amp;gt;gpuwide&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;QOS&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for each of the other QOSes&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
= Job priority =&lt;br /&gt;
&lt;br /&gt;
Once the execution of a job has been requested, the job is not run immediately: it is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039; by SLURM, together with all the other jobs awaiting execution. The job on top of the queue at any time is the first to be put into execution as soon as the resources it requires are available. &lt;br /&gt;
&lt;br /&gt;
The order of the items in the job queue depends on their &amp;#039;&amp;#039;&amp;#039;priority&amp;#039;&amp;#039;&amp;#039;. The job with the highest priority is put by SLURM on top of the queue, while all other jobs are put in the queue in order of descending priority.&lt;br /&gt;
&lt;br /&gt;
The goal of SLURM is to maximise resource availability: i.e., to ensure the shorter possible wait time before job execution. To achieve this goal, in Mufasa SLURM is configured to &amp;#039;&amp;#039;&amp;#039;encourage users not to ask for resources or execution time that their job doesn&amp;#039;t need&amp;#039;&amp;#039;&amp;#039;. This is done via the priority mechanism: the more resources and/or time a job requests, the lower its priority will be; and the later it will be executed.&lt;br /&gt;
&lt;br /&gt;
Priority management in Mufasa is designed to set up a &amp;#039;&amp;#039;&amp;#039;virtuous cycle&amp;#039;&amp;#039;&amp;#039; where users, by carefully choosing what to ask for, obtain two results:&lt;br /&gt;
* they ensure that their job is executed as soon as possible;&lt;br /&gt;
* they leave as much as possible of Mufasa&amp;#039;s resources free for other users&amp;#039;s jobs.&lt;br /&gt;
&lt;br /&gt;
== Elements determining job priority ==&lt;br /&gt;
In Mufasa, the priority of a job is computed by SLURM according to the following elements:&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#research users and students users|User category]]&amp;#039;&amp;#039;&amp;#039; (i.e., &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;)&lt;br /&gt;
::: Used to provide higher priority to jobs run by &amp;#039;&amp;#039;&amp;#039;research personnel&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#SLURM Quality of Service (QOS)|QOS]]&amp;#039;&amp;#039;&amp;#039; used by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;less resources&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Number of CPUs&amp;#039;&amp;#039;&amp;#039; requested by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;a lower number of CPUs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job Age&amp;#039;&amp;#039;&amp;#039;, i.e. the time that the job has been waiting in the queue&lt;br /&gt;
::: Used to provide higher priority to jobs which have been &amp;#039;&amp;#039;&amp;#039;queued for a longer time&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;FairShare&amp;#039;&amp;#039;&amp;#039;, i.e. a factor computed by SLURM to balance use of the system by different users&lt;br /&gt;
::: Used to provide higher priority to jobs by users who &amp;#039;&amp;#039;&amp;#039;used Mufasa&amp;#039;s resources less than others&amp;#039;&amp;#039;&amp;#039; (see [[#How FairShare works|below]])&lt;br /&gt;
&lt;br /&gt;
You can understand how each of these elements contribute to the priority of your jobs using command &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;. A tailored version of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; that only displays the parameters listed above, in the same order, is the following:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sprio -o &amp;quot;%6i %9u %.5Y %.10B %.5Q %.9T %.5A %.5F&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
Here is an example of its output:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
JOBID  USER      PRIOR      ASSOC   QOS      TRES   AGE FAIRS&lt;br /&gt;
5774   itombini   3492       2500   750  cpu=-800   906   136&lt;br /&gt;
5780   fbottazzi  2266        625   750  cpu=-600   420  1071&lt;br /&gt;
5783   mzoccola   1766        625   750  cpu=-800   217   974&lt;br /&gt;
5790   amoglia    2717       2500   375  cpu=-800   156   487&lt;br /&gt;
5791   qliu       2232       2500   375  cpu=-800   138    19&lt;br /&gt;
5792   fpastori   2197       2500   375  cpu=-800    84    39&lt;br /&gt;
5793   bagostini  2034        625  1500 cpu=-1200    18  1091&lt;br /&gt;
5794   lpassoni   2517       2500   750  cpu=-800     9    58&lt;br /&gt;
5795   lpassoni   2517       2500   750  cpu=-800     9    58&lt;br /&gt;
5796   adurat     3101       2500   750  cpu=-800     8   643&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
In it, column ASSOC (i.e., association) corresponds to user category, while column TRES corresponds to the number of CPUs. Note that the contribution of CPUs to priority is &amp;#039;&amp;#039;negative&amp;#039;&amp;#039;, i.e. it is a penalty. Column PRIOR provides the overall priority of each job.&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Important note&lt;br /&gt;
:: The output of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; is only relevant for jobs &amp;#039;&amp;#039;&amp;#039;waiting to be executed&amp;#039;&amp;#039;&amp;#039;. For jobs already in execution, value shown by &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; is the priority that the jobs would get if it was still queued, which usually does not correspond to the actual priority.&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
=== How FairShare works ===&lt;br /&gt;
In Mufasa 2.0, the FairShare of each user is computed according to these rules:&lt;br /&gt;
# It considers the quantity used by (current and past) user jobs of:&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;CPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of CPUs used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;RAM&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the amount of RAM used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;GPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of GPUs used&lt;br /&gt;
#** note: 40 GB GPUs have double the impact on FairShare than 20 GB GPUs&lt;br /&gt;
# The impact on FairShare of any use of resources is &amp;#039;&amp;#039;&amp;#039;proportional to its duration&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
#* example: using 32 GB of RAM for 48h has the same impact of using 64 GB of RAM for 24h&lt;br /&gt;
# FairShare has a &amp;quot;fading memory&amp;quot;: i.e., resource use has less and less impact on FairShare the farther in the past it is&lt;br /&gt;
#* consequence: over time, the &amp;quot;history&amp;quot; of a user gets forgotten by FairShare&lt;br /&gt;
&lt;br /&gt;
== How to maximise the priority of your jobs ==&lt;br /&gt;
&lt;br /&gt;
Every time you run a SLURM job, follow these guidelines:&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Choose the less powerful QOS compatible with the needs of your job&lt;br /&gt;
:: QOSes with access to less resources have higher base priority&lt;br /&gt;
&lt;br /&gt;
; Only request CPUs and RAM that your job really needs&lt;br /&gt;
:: Usually, code exploits multiple CPUs only if specifically designed to do so: if unsure, only ask for 1 CPU&lt;br /&gt;
:: The fewer resources your job asks for, the less it will wait before they become available&lt;br /&gt;
:: Asking for fewer CPUs and/or less RAM improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Do not request more time than your jobs needs to complete&lt;br /&gt;
:: Make a worst-case estimate and only ask for that duration&lt;br /&gt;
:: Asking for less of Mufasa&amp;#039;s time improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Debug and test code using less powerful QOSes before running it with more powerful QOSes&lt;br /&gt;
:: Your test jobs will get a higher priority and your FairShare will improve&lt;br /&gt;
&lt;br /&gt;
; Cancel jobs that you don&amp;#039;t need them anymore&lt;br /&gt;
:: Always use [[User_Jobs#Cancelling_a_job_with_scancel|scancel]] to delete completed (or crashed) jobs: your Fairshare will improve&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Protip: it&amp;#039;s a good idea to [[User Jobs#Looking for unused GPUs|check for unused GPUs]] before choosing what to request. Requesting a GPU that is currently idle will help your job get executed sooner.&lt;br /&gt;
&lt;br /&gt;
= SLURM partitions =&lt;br /&gt;
&lt;br /&gt;
Partitions are another mechanism provided by SLURM to create different levels of access to system resources. Since in Mufasa 2.0 access to resources is controlled via [[#SLURM Quality of Service (QOS)|QOSes]], partitions are not very relevant. (This is one of the main differences between Mufasa 1.0 and Mufasa 2.0.)&lt;br /&gt;
&lt;br /&gt;
Note, however, that the default values for some features of SLURM jobs (e.g., duration) are [[#Default values|set by the partition]].&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, there is a single SLURM partition, called &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt;, and all jobs run on it. The state of &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt; can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sinfo -o &amp;quot;%10P %5a %9T %11L %10l&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PARTITION  AVAIL STATE     DEFAULTTIME TIMELIMIT &lt;br /&gt;
jobs*      up    idle      1:00:00     3-00:00:00&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where columns correspond to the following information:&lt;br /&gt;
&lt;br /&gt;
:; PARTITION&lt;br /&gt;
:: name of the partition; the asterisks indicates that it&amp;#039;s the default one&lt;br /&gt;
&lt;br /&gt;
:; AVAIL&lt;br /&gt;
:: state/availability of the partition: see [[#Partition availability|below]]&lt;br /&gt;
&lt;br /&gt;
:; STATE&lt;br /&gt;
:: state (using [https://slurm.schedmd.com/sinfo.html#SECTION_NODE-STATE-CODES these codes])&lt;br /&gt;
:: typical values are &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;mixed&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that some of the resources are busy executing jobs while other are idle, and &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;allocated&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that all of the resources are in use&lt;br /&gt;
&lt;br /&gt;
:; DEFAULTTIME&lt;br /&gt;
:: default runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:; TIMELIMIT&lt;br /&gt;
:: maximum runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
The asterisk at the end of the partition name indicates the default partition, i.e. the one on which jobs which do not ask for a specific partition are run.&lt;br /&gt;
&lt;br /&gt;
Command &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; does not tell you about the &amp;#039;&amp;#039;jobs&amp;#039;&amp;#039; submitted to a partition. This information is obtained, instead, with [[User Jobs#Inspecting jobs with squeue|command &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;]].&lt;br /&gt;
&lt;br /&gt;
== Partition availability ==&lt;br /&gt;
&lt;br /&gt;
The most important information that &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; provides is the &amp;#039;&amp;#039;&amp;#039;availability&amp;#039;&amp;#039;&amp;#039; (also called &amp;#039;&amp;#039;&amp;#039;state&amp;#039;&amp;#039;&amp;#039;) of partitions. This is shown in column &amp;quot;AVAIL&amp;quot;. Possible partition states are:&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is available&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed as soon as resources allow&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is in the process of becoming unavailable (i.e., of entering the &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; state: see below)&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is unavailable&lt;br /&gt;
:: There are no running jobs&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
When a partition goes from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; no harm is done to running jobs. In a normally functioning SLURM system, the passage from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; happens only when no jobs are running on the partition. If (e.g., due to a malfunction) the passage happens with jobs still running, they get killed.&lt;br /&gt;
&lt;br /&gt;
A partition in state &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; requires intervention by a [[Roles|Job Administrator]] to be restored to &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
== Default values ==&lt;br /&gt;
&lt;br /&gt;
The features of SLURM partitions, including the &amp;#039;&amp;#039;&amp;#039;default values&amp;#039;&amp;#039;&amp;#039; which are applied to jobs that do not make explicit requests, can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scontrol show partition&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PartitionName=jobs&lt;br /&gt;
   AllowGroups=ALL AllowAccounts=ALL AllowQos=nogpu,gpulight,gpu,gpuwide,gpuheavy-20,gpuheavy-40&lt;br /&gt;
   AllocNodes=ALL Default=YES QoS=N/A&lt;br /&gt;
=&amp;gt; DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO&lt;br /&gt;
   MaxNodes=UNLIMITED MaxTime=3-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED&lt;br /&gt;
   Nodes=gn01&lt;br /&gt;
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO&lt;br /&gt;
   OverTimeLimit=NONE PreemptMode=OFF&lt;br /&gt;
   State=UP TotalCPUs=48 TotalNodes=1 SelectTypeParameters=NONE&lt;br /&gt;
   JobDefaults=(null)&lt;br /&gt;
=&amp;gt; DefMemPerNode=4096 MaxMemPerNode=UNLIMITED&lt;br /&gt;
   TRES=cpu=48,mem=1011435M,node=1,billing=49,gres/gpu=13,gres/gpu:3g.20gb=5,gres/gpu:40gb=3,gres/gpu:4g.20gb=5&lt;br /&gt;
   TRESBillingWeights=cpu=1.0,gres/gpu:3g.20gb=6.0,gres/gpu:4g.20gb=6.0,gres/gpu:40gb=6.0,mem=0.05g&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In the example, we have highlighted with  &amp;#039;&amp;#039;&amp;#039;=&amp;gt;&amp;#039;&amp;#039;&amp;#039;  the most relevant default values for Mufasa users, i.e.:&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefaultTime&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default execution time assigned to a job run on the partition (e.g., 1 hour)&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefMemPerNode&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default amount of RAM assigned to a job run on the partition (e.g., 4GB)&lt;br /&gt;
&lt;br /&gt;
= System resources subjected to limitations =&lt;br /&gt;
&lt;br /&gt;
In systems based on SLURM like Mufasa, &amp;#039;&amp;#039;&amp;#039;TRES (Trackable RESources)&amp;#039;&amp;#039;&amp;#039; are (from [https://slurm.schedmd.com/tres.html SLURM&amp;#039;s documentation] &amp;quot;&amp;#039;&amp;#039;resources that can be tracked for usage or used to enforce limits against.&amp;#039;&amp;#039;&amp;quot;&lt;br /&gt;
&lt;br /&gt;
TRES include CPUs, RAM and &amp;#039;&amp;#039;&amp;#039;GRES&amp;#039;&amp;#039;&amp;#039;. The last term stands for &amp;#039;&amp;#039;Generic RESources&amp;#039;&amp;#039; that a job may need for its execution. In Mufasa, the only &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; resources are the GPUs.&lt;br /&gt;
&lt;br /&gt;
== &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax ==&lt;br /&gt;
&lt;br /&gt;
To ask SLURM to assign GRES resources (i.e., GPUs) to a job, a special syntax must be used. Precisely, the name of each GPU resource takes the form&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:Name:Type&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
Considering the [[System#CPUs and GPUs|GPU complement of Mufasa]], &amp;lt;code&amp;gt;Type&amp;lt;/code&amp;gt; takes the following values:&lt;br /&gt;
&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:40gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 40 Gbytes of RAM&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:4g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 4 compute units&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 3 compute units&lt;br /&gt;
&lt;br /&gt;
So, for instance,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
identifies a resource corresponding to a GPU with 20 GB of RAM and 3 compute units.&lt;br /&gt;
&lt;br /&gt;
When asking for a GRES resource (e.g., in an &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command or an &amp;lt;code&amp;gt;SBATCH&amp;lt;/code&amp;gt; directive of an [[User Jobs#Using execution scripts to run jobs|execution script]]), the syntax required by SLURM is&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:&amp;lt;Type&amp;gt;:&amp;lt;Quantity&amp;gt;&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
where &amp;lt;code&amp;gt;Quantity&amp;lt;/code&amp;gt; is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of type &amp;lt;code&amp;gt;4g.20gb&amp;lt;/code&amp;gt; the syntax is&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:4g.20gb:2&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SLURM&amp;#039;s &amp;#039;&amp;#039;generic resources&amp;#039;&amp;#039; are defined in &amp;lt;code&amp;gt;/etc/slurm/gres.conf&amp;lt;/code&amp;gt;. In order to make GPUs available to SLURM&amp;#039;s &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; management, Mufasa makes use of Nvidia&amp;#039;s [https://developer.nvidia.com/nvidia-management-library-nvml NVML library]. For additional information see [https://slurm.schedmd.com/gres.html SLURM&amp;#039;s documentation].&lt;/div&gt;</summary>
		<author><name>GiulioFontana</name></author>
	</entry>
	<entry>
		<id>https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2569</id>
		<title>SLURM</title>
		<link rel="alternate" type="text/html" href="https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2569"/>
		<updated>2026-11-03T14:30:18Z</updated>

		<summary type="html">&lt;p&gt;GiulioFontana: /* Elements determining job priority */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page presents the features of SLURM that are most relevant to Mufasa&amp;#039;s [[Roles|Job Users]]. Job Users can submit jobs for execution, cancel their own jobs, and see other users&amp;#039; jobs (but not intervene on them).&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa &amp;#039;&amp;#039;&amp;#039;must use SLURM&amp;#039;&amp;#039;&amp;#039; to run resource-heavy processes, i.e. computing jobs that require one or more of the following:&lt;br /&gt;
* GPUs&lt;br /&gt;
* multiple CPUs&lt;br /&gt;
* powerful CPUs&lt;br /&gt;
* a significant amount of RAM&lt;br /&gt;
&lt;br /&gt;
In fact, only processes run via SLURM have access to all the resources of Mufasa. Processes run outside SLURM are executed by the [[System#Login server|login server]] virtual machine, which has minimal resources and no GPUs. Using SLURM is therefore the only way to execute resource-heavy jobs on Mufasa (this is a key difference between Mufasa 1.0 and Mufasa 2.0).&lt;br /&gt;
&lt;br /&gt;
= SLURM in a nutshell =&lt;br /&gt;
&lt;br /&gt;
Computation jobs on Mufasa needs to be launched via [[System#The SLURM job scheduling system|SLURM]]. SLURM provides jobs with access to the [[#System resources subjected to limitations|physical resources]] of Mufasa, such as CPUs, GPUs and RAM. Thanks to SLURM, processing jobs share system resources, optimising their occupation and availability. &lt;br /&gt;
&lt;br /&gt;
When a user runs a job, the job does not get executed immediately and is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039;. SLURM executes jobs according to their order in the queue: the top job in the queue gets executed as soon as the necessary resources are available, while jobs lower in the queue wait longer. The position of a job in the queue is due to the &amp;#039;&amp;#039;&amp;#039;[[#Job priority|priority]]&amp;#039;&amp;#039;&amp;#039; assigned to it by SLURM, with higher-priority jobs closer to the top. As a general rule:&lt;br /&gt;
&lt;br /&gt;
;: &amp;#039;&amp;#039;&amp;#039;the greater the fraction of Mufasa&amp;#039;s overall resources that a job asks for, the lower the job&amp;#039;s priority will be&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
The priority mechanism is used to encourage users to use Mufasa&amp;#039;s resources (i.e.: GPUs, CPUs, RAM, execution time) in an effective and equitable manner. This page includes a [[#How_to_maximise_the_priority_of_your_jobs|chart explaining how to maximise the priority of your jobs]].&lt;br /&gt;
&lt;br /&gt;
The &amp;#039;&amp;#039;&amp;#039;time&amp;#039;&amp;#039;&amp;#039; available to a job for its execution is controlled by SLURM. When a user requests execution of a job, they must specify the duration of the time slot that the job needs. The job must complete its execution before the end of the requested time slot, because at the end of such slot it gets killed by SLURM.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0 access to system resources is managed via SLURM&amp;#039;s [[#SLURM Quality of Service (QOS)|Quality of Service (QOS)]] mechanism (Mufasa 1.0 used [[#SLURM_partitions|partitions]] instead). While [[User Jobs#Running jobs with SLURM|launching a processing job via SLURM]], the user must always specify the chosen QOS. QOSes differ in the set of resources that they provide access to because each of them is designed to fit a given type of job.&lt;br /&gt;
&lt;br /&gt;
Mufasa sets limits to the number of jobs by the same user. This page includes a [[#Limits on jobs by the same user|table summarising such limits]].&lt;br /&gt;
&lt;br /&gt;
= SLURM Quality of Service (QOS) =&lt;br /&gt;
&lt;br /&gt;
Through &amp;#039;&amp;#039;&amp;#039;Quality of Services&amp;#039;&amp;#039;&amp;#039; (&amp;#039;&amp;#039;&amp;#039;QOSes&amp;#039;&amp;#039;&amp;#039;), SLURM lets system configurators assign a name to a set of related constraints.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, QOSes are used to define different levels of access to the server&amp;#039;s resources. When [[User Jobs|executing a job with SLURM]], a user must always &amp;#039;&amp;#039;&amp;#039;specify the QOS&amp;#039;&amp;#039;&amp;#039; that their job will use: this choice, in turn, determines what resources the job is able to access and influences the [[#Job priority|priority]] of the job.&lt;br /&gt;
&lt;br /&gt;
Mufasa&amp;#039;s QOSes and their features can be inspected with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list qos format=name%-11,priority,MaxSubmit,maxwall,maxtres%-80,mintres%-18&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following (see below for the same data in table form):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
Name          Priority MaxSubmit     MaxWall MaxTRES                                                                          MinTRES            &lt;br /&gt;
----------- ---------- --------- ----------- -------------------------------------------------------------------------------- ------------------ &lt;br /&gt;
normal               0                                                                                                                           &lt;br /&gt;
nogpu                4         1  3-00:00:00 cpu=16,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=128G                    &lt;br /&gt;
gpuheavy-20          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=2,mem=128G             gres/gpu:4g.20gb=1 &lt;br /&gt;
gpuheavy-40          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=1,gres/gpu:4g.20gb=0,mem=128G             gres/gpu:40gb=1    &lt;br /&gt;
gpulight             8         1    12:00:00 cpu=2,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpu                  2         1  1-00:00:00 cpu=8,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpuwide              2         2  1-00:00:00 cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=1,mem=64G              gres/gpu:4g.20gb=1 &lt;br /&gt;
build               32         1    02:00:00 cpu=2,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=16G&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The columns of this output are the following:&lt;br /&gt;
&lt;br /&gt;
:; Name&lt;br /&gt;
:: name of the QOS&lt;br /&gt;
&lt;br /&gt;
:; Priority&lt;br /&gt;
:: priority tier associated to the QOS (higher value = higher priority): see [[#Job priority|Job priority]] for details&lt;br /&gt;
&lt;br /&gt;
:; MaxSubmit&lt;br /&gt;
:: maximum number of jobs from a single user that can be submitted to SLURM with this QOS; submitted jobs include both running and queued jobs&lt;br /&gt;
:: See [[#Limits on jobs by the same user|Limits on jobs by the same user]] for an overview of the limits on jobs set by Mufasa.&lt;br /&gt;
&lt;br /&gt;
:; MaxWall&lt;br /&gt;
:: maximum wall clock duration of the jobs using the QOS (after which they are killed by SLURM), in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
:: For some QOSes these are not set: it means that they are determined by the [[#SLURM partitions|partition]]. Partitions also define the [[#Default values|default duration]] of jobs.&lt;br /&gt;
&lt;br /&gt;
:; MaxTRES&lt;br /&gt;
:: amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) available to a job using the QOS, where&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;cpu=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of CPUs (i.e., processor cores) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of CPUs specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;gres/&amp;#039;&amp;#039;gpu:Type&amp;#039;&amp;#039;=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of GPUs of class &amp;lt;code&amp;gt;&amp;#039;&amp;#039;Type&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; (see [[User Jobs#gres syntax|&amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax]]) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; (for QOSes that allow access to GPUs) if not specified, the job cannot be launched&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;mem=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;G&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum amount of system RAM is &amp;#039;&amp;#039;K&amp;#039;&amp;#039; GBytes&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of RAM specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
&lt;br /&gt;
:; MinTRES&lt;br /&gt;
:: minimum amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) that a job using the QOS must request in order to actually get executed by SLURM.&lt;br /&gt;
:: (If your job does not actually need these resources, you&amp;#039;ve chosen the wrong QOS: you can use one with a higher priority.)&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; QOS is the default one, and exists only to ensure that users always specify a QOS when running a job. Since &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; has zero priority and no resources, a job run using this QOS would never be run.&lt;br /&gt;
&lt;br /&gt;
The information provided by the &amp;lt;code&amp;gt;sacctmgr list qos&amp;lt;/code&amp;gt; command above is summarised by the following table:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! QOS&lt;br /&gt;
! Priority&amp;lt;/br&amp;gt;tier&lt;br /&gt;
! Max&amp;lt;/br&amp;gt;Submit&lt;br /&gt;
! MaxWall&amp;lt;/br&amp;gt;[h]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;CPUs&lt;br /&gt;
! max RAM&amp;lt;/br&amp;gt;[GB]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;3g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;4g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;40GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | build&lt;br /&gt;
| 32&lt;br /&gt;
| 1&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 16&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpulight&lt;br /&gt;
| 8&lt;br /&gt;
| 1&lt;br /&gt;
| 12&lt;br /&gt;
| 2&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | nogpu&lt;br /&gt;
| 4&lt;br /&gt;
| 1&lt;br /&gt;
| 72&lt;br /&gt;
| 16&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpu&lt;br /&gt;
| 2&lt;br /&gt;
| 1&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuwide&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-20&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| 2 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-40&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
A key piece of information in the table above is the &amp;#039;&amp;#039;&amp;#039;priority tier&amp;#039;&amp;#039;&amp;#039; associated to each QOS. In Mufasa 2.0, priority tiers are used to encourage users to use the &amp;#039;&amp;#039;&amp;#039;least powerful QOS that is compatible with their needs&amp;#039;&amp;#039;&amp;#039;, where &amp;quot;powerful&amp;quot; means &amp;quot;rich with resources&amp;quot;. Less powerful QOSes increase the priority of the jobs that use them, so these jobs tend to be executed sooner. See [[#Job priority|Job priority]] for details about how priority affects the execution order of jobs in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
== The &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS ==&lt;br /&gt;
&lt;br /&gt;
This QOS is specifically designed to be used by Mufasa users to &amp;#039;&amp;#039;&amp;#039;quickly build [[System#Containers|container images]]&amp;#039;&amp;#039;&amp;#039;. Its associated priority tier is very high: this, combined with the fact that jobs using this QOS require few resources, means that such jobs usually get executed very soon.&lt;br /&gt;
&lt;br /&gt;
On the other hand, the limited resources and the lack of access to the GPUs make the &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS unsuitable for other tasks.&lt;br /&gt;
&lt;br /&gt;
See [[Singularity#Building Singularity images|Building Singularity images]] for directions about building Singularity container images.&lt;br /&gt;
&lt;br /&gt;
== Restricted QOSes ==&lt;br /&gt;
&lt;br /&gt;
In Mufasa, the most powerful QOSes are reserved to researchers (called &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users: these include academic personnel and Ph.D. students). M.Sc. students, called &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users, cannot use them while running jobs.&lt;br /&gt;
&lt;br /&gt;
See [[#research users and students users|below]] to understand how &amp;#039;&amp;#039;user categories&amp;#039;&amp;#039; work in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
= &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users =&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa belong to two &amp;#039;&amp;#039;&amp;#039;user categories&amp;#039;&amp;#039;&amp;#039;. Categories provide users with different access to  to Mufasa&amp;#039;s resources: the idea being to provide researchers with more access while still letting students benefit from the server&amp;#039;s capabilities.&lt;br /&gt;
&lt;br /&gt;
User categories in Mufasa are the following:&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. academic personnel and Ph.D. students&lt;br /&gt;
::: * can use all QOSes, including the the [[#Restricted QOSes|restricted ones]]&lt;br /&gt;
::: * their jobs have a higher &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a higher number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. M.Sc. students&lt;br /&gt;
::: * cannot use the [[#Restricted QOSes|restricted QOSes]]&lt;br /&gt;
::: * their jobs have a lower &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a lower number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
You can inspect the differences in priority and running jobs between &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association format=&amp;quot;account,priority,maxjobs&amp;quot; | grep -E &amp;#039;Account|research|students&amp;#039;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
   Account   Priority MaxJobs &lt;br /&gt;
  research          4       2 &lt;br /&gt;
  students          1       1&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
To know what limits apply to your own user, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association where user=$USER format=&amp;quot;user,priority,maxjobs,qos%-60&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
      User   Priority MaxJobs QOS                                                          &lt;br /&gt;
---------- ---------- ------- ------------------------------------------------------------ &lt;br /&gt;
    preali          4       2 build,gpu,gpuheavy-20,gpuheavy-40,gpulight,gpuwide,nogpu&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The list under &amp;quot;QOS&amp;quot; shows what QOSes your user is allowed to use when [[User Jobs|running jobs]]. &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users can use all of them, while &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users can only access a subset of them.&lt;br /&gt;
&lt;br /&gt;
= Limits on jobs by the same user =&lt;br /&gt;
&lt;br /&gt;
Mufasa enforces limits on the number of jobs from a single user. Such limits aim at preventing users from &amp;quot;hogging&amp;quot; system resources, and apply to:&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;submitted jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that the user asked SLURM to execute, each of which may currently be either running or queued&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;running jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that are currently in execution&lt;br /&gt;
&lt;br /&gt;
Limits on submitted jobs are set via [[#SLURM Quality of Service (QOS)|QOSes]], while limits on running jobs are set via [[#research users and students users|user category]].&lt;br /&gt;
&lt;br /&gt;
The following table summarises what limits exist:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! &lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;running&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;submitted&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | global limits&amp;lt;br/&amp;gt;(system-wide)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to limits set by individual QOSes (below)&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | limits for&amp;lt;br/&amp;gt;individual&amp;lt;/br&amp;gt;QOSes&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to global limits (above)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for the&amp;#039;&amp;#039; &amp;lt;code&amp;gt;gpuwide&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;QOS&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for each of the other QOSes&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
= Job priority =&lt;br /&gt;
&lt;br /&gt;
Once the execution of a job has been requested, the job is not run immediately: it is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039; by SLURM, together with all the other jobs awaiting execution. The job on top of the queue at any time is the first to be put into execution as soon as the resources it requires are available. &lt;br /&gt;
&lt;br /&gt;
The order of the items in the job queue depends on their &amp;#039;&amp;#039;&amp;#039;priority&amp;#039;&amp;#039;&amp;#039;. The job with the highest priority is put by SLURM on top of the queue, while all other jobs are put in the queue in order of descending priority.&lt;br /&gt;
&lt;br /&gt;
The goal of SLURM is to maximise resource availability: i.e., to ensure the shorter possible wait time before job execution. To achieve this goal, in Mufasa SLURM is configured to &amp;#039;&amp;#039;&amp;#039;encourage users not to ask for resources or execution time that their job doesn&amp;#039;t need&amp;#039;&amp;#039;&amp;#039;. This is done via the priority mechanism: the more resources and/or time a job requests, the lower its priority will be; and the later it will be executed.&lt;br /&gt;
&lt;br /&gt;
Priority management in Mufasa is designed to set up a &amp;#039;&amp;#039;&amp;#039;virtuous cycle&amp;#039;&amp;#039;&amp;#039; where users, by carefully choosing what to ask for, obtain two results:&lt;br /&gt;
* they ensure that their job is executed as soon as possible;&lt;br /&gt;
* they leave as much as possible of Mufasa&amp;#039;s resources free for other users&amp;#039;s jobs.&lt;br /&gt;
&lt;br /&gt;
== Elements determining job priority ==&lt;br /&gt;
In Mufasa, the priority of a job is computed by SLURM according to the following elements:&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#research users and students users|User category]]&amp;#039;&amp;#039;&amp;#039; (i.e., &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;)&lt;br /&gt;
::: Used to provide higher priority to jobs run by &amp;#039;&amp;#039;&amp;#039;research personnel&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#SLURM Quality of Service (QOS)|QOS]]&amp;#039;&amp;#039;&amp;#039; used by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;less resources&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Number of CPUs&amp;#039;&amp;#039;&amp;#039; requested by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;a lower number of CPUs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job Age&amp;#039;&amp;#039;&amp;#039;, i.e. the time that the job has been waiting in the queue&lt;br /&gt;
::: Used to provide higher priority to jobs which have been &amp;#039;&amp;#039;&amp;#039;queued for a longer time&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;FairShare&amp;#039;&amp;#039;&amp;#039;, i.e. a factor computed by SLURM to balance use of the system by different users&lt;br /&gt;
::: Used to provide higher priority to jobs by users who &amp;#039;&amp;#039;&amp;#039;used Mufasa&amp;#039;s resources less than others&amp;#039;&amp;#039;&amp;#039; (see [[#How FairShare works|below]])&lt;br /&gt;
&lt;br /&gt;
You can understand how each of these elements contribute to the priority of your jobs using command &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;. A tailored version of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; that only displays the parameters listed above, in the same order, is the following:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sprio -o &amp;quot;%6i %9u %.5Y %.10B %.5Q %.9T %.5A %.5F&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
Here is an example of its output:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
JOBID  USER      PRIOR      ASSOC   QOS      TRES   AGE FAIRS&lt;br /&gt;
5774   itombini   3492       2500   750  cpu=-800   906   136&lt;br /&gt;
5780   fbottazzi  2266        625   750  cpu=-600   420  1071&lt;br /&gt;
5783   mzoccola   1766        625   750  cpu=-800   217   974&lt;br /&gt;
5790   amoglia    2717       2500   375  cpu=-800   156   487&lt;br /&gt;
5791   qliu       2232       2500   375  cpu=-800   138    19&lt;br /&gt;
5792   fpastori   2197       2500   375  cpu=-800    84    39&lt;br /&gt;
5793   bagostini  2034        625  1500 cpu=-1200    18  1091&lt;br /&gt;
5794   lpassoni   2517       2500   750  cpu=-800     9    58&lt;br /&gt;
5795   lpassoni   2517       2500   750  cpu=-800     9    58&lt;br /&gt;
5796   adurat     3101       2500   750  cpu=-800     8   643&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
In it, column ASSOC (i.e., association) corresponds to user category, while column TRES corresponds to the number of CPUs. Note that the contribution of CPUs to priority is &amp;#039;&amp;#039;negative&amp;#039;&amp;#039;, i.e. it is a penalty. Column PRIOR provides the overall priority of each job.&lt;br /&gt;
&lt;br /&gt;
=== How FairShare works ===&lt;br /&gt;
In Mufasa 2.0, the FairShare of each user is computed according to these rules:&lt;br /&gt;
# It considers the quantity used by (current and past) user jobs of:&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;CPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of CPUs used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;RAM&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the amount of RAM used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;GPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of GPUs used&lt;br /&gt;
#** note: 40 GB GPUs have double the impact on FairShare than 20 GB GPUs&lt;br /&gt;
# The impact on FairShare of any use of resources is &amp;#039;&amp;#039;&amp;#039;proportional to its duration&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
#* example: using 32 GB of RAM for 48h has the same impact of using 64 GB of RAM for 24h&lt;br /&gt;
# FairShare has a &amp;quot;fading memory&amp;quot;: i.e., resource use has less and less impact on FairShare the farther in the past it is&lt;br /&gt;
#* consequence: over time, the &amp;quot;history&amp;quot; of a user gets forgotten by FairShare&lt;br /&gt;
&lt;br /&gt;
== How to maximise the priority of your jobs ==&lt;br /&gt;
&lt;br /&gt;
Every time you run a SLURM job, follow these guidelines:&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Choose the less powerful QOS compatible with the needs of your job&lt;br /&gt;
:: QOSes with access to less resources have higher base priority&lt;br /&gt;
&lt;br /&gt;
; Only request CPUs and RAM that your job really needs&lt;br /&gt;
:: Usually, code exploits multiple CPUs only if specifically designed to do so: if unsure, only ask for 1 CPU&lt;br /&gt;
:: The fewer resources your job asks for, the less it will wait before they become available&lt;br /&gt;
:: Asking for fewer CPUs and/or less RAM improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Do not request more time than your jobs needs to complete&lt;br /&gt;
:: Make a worst-case estimate and only ask for that duration&lt;br /&gt;
:: Asking for less of Mufasa&amp;#039;s time improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Debug and test code using less powerful QOSes before running it with more powerful QOSes&lt;br /&gt;
:: Your test jobs will get a higher priority and your FairShare will improve&lt;br /&gt;
&lt;br /&gt;
; Cancel jobs that you don&amp;#039;t need them anymore&lt;br /&gt;
:: Always use [[User_Jobs#Cancelling_a_job_with_scancel|scancel]] to delete completed (or crashed) jobs: your Fairshare will improve&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Protip: it&amp;#039;s a good idea to [[User Jobs#Looking for unused GPUs|check for unused GPUs]] before choosing what to request. Requesting a GPU that is currently idle will help your job get executed sooner.&lt;br /&gt;
&lt;br /&gt;
= SLURM partitions =&lt;br /&gt;
&lt;br /&gt;
Partitions are another mechanism provided by SLURM to create different levels of access to system resources. Since in Mufasa 2.0 access to resources is controlled via [[#SLURM Quality of Service (QOS)|QOSes]], partitions are not very relevant. (This is one of the main differences between Mufasa 1.0 and Mufasa 2.0.)&lt;br /&gt;
&lt;br /&gt;
Note, however, that the default values for some features of SLURM jobs (e.g., duration) are [[#Default values|set by the partition]].&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, there is a single SLURM partition, called &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt;, and all jobs run on it. The state of &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt; can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sinfo -o &amp;quot;%10P %5a %9T %11L %10l&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PARTITION  AVAIL STATE     DEFAULTTIME TIMELIMIT &lt;br /&gt;
jobs*      up    idle      1:00:00     3-00:00:00&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where columns correspond to the following information:&lt;br /&gt;
&lt;br /&gt;
:; PARTITION&lt;br /&gt;
:: name of the partition; the asterisks indicates that it&amp;#039;s the default one&lt;br /&gt;
&lt;br /&gt;
:; AVAIL&lt;br /&gt;
:: state/availability of the partition: see [[#Partition availability|below]]&lt;br /&gt;
&lt;br /&gt;
:; STATE&lt;br /&gt;
:: state (using [https://slurm.schedmd.com/sinfo.html#SECTION_NODE-STATE-CODES these codes])&lt;br /&gt;
:: typical values are &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;mixed&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that some of the resources are busy executing jobs while other are idle, and &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;allocated&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that all of the resources are in use&lt;br /&gt;
&lt;br /&gt;
:; DEFAULTTIME&lt;br /&gt;
:: default runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:; TIMELIMIT&lt;br /&gt;
:: maximum runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
The asterisk at the end of the partition name indicates the default partition, i.e. the one on which jobs which do not ask for a specific partition are run.&lt;br /&gt;
&lt;br /&gt;
Command &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; does not tell you about the &amp;#039;&amp;#039;jobs&amp;#039;&amp;#039; submitted to a partition. This information is obtained, instead, with [[User Jobs#Inspecting jobs with squeue|command &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;]].&lt;br /&gt;
&lt;br /&gt;
== Partition availability ==&lt;br /&gt;
&lt;br /&gt;
The most important information that &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; provides is the &amp;#039;&amp;#039;&amp;#039;availability&amp;#039;&amp;#039;&amp;#039; (also called &amp;#039;&amp;#039;&amp;#039;state&amp;#039;&amp;#039;&amp;#039;) of partitions. This is shown in column &amp;quot;AVAIL&amp;quot;. Possible partition states are:&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is available&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed as soon as resources allow&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is in the process of becoming unavailable (i.e., of entering the &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; state: see below)&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is unavailable&lt;br /&gt;
:: There are no running jobs&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
When a partition goes from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; no harm is done to running jobs. In a normally functioning SLURM system, the passage from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; happens only when no jobs are running on the partition. If (e.g., due to a malfunction) the passage happens with jobs still running, they get killed.&lt;br /&gt;
&lt;br /&gt;
A partition in state &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; requires intervention by a [[Roles|Job Administrator]] to be restored to &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
== Default values ==&lt;br /&gt;
&lt;br /&gt;
The features of SLURM partitions, including the &amp;#039;&amp;#039;&amp;#039;default values&amp;#039;&amp;#039;&amp;#039; which are applied to jobs that do not make explicit requests, can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scontrol show partition&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PartitionName=jobs&lt;br /&gt;
   AllowGroups=ALL AllowAccounts=ALL AllowQos=nogpu,gpulight,gpu,gpuwide,gpuheavy-20,gpuheavy-40&lt;br /&gt;
   AllocNodes=ALL Default=YES QoS=N/A&lt;br /&gt;
=&amp;gt; DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO&lt;br /&gt;
   MaxNodes=UNLIMITED MaxTime=3-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED&lt;br /&gt;
   Nodes=gn01&lt;br /&gt;
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO&lt;br /&gt;
   OverTimeLimit=NONE PreemptMode=OFF&lt;br /&gt;
   State=UP TotalCPUs=48 TotalNodes=1 SelectTypeParameters=NONE&lt;br /&gt;
   JobDefaults=(null)&lt;br /&gt;
=&amp;gt; DefMemPerNode=4096 MaxMemPerNode=UNLIMITED&lt;br /&gt;
   TRES=cpu=48,mem=1011435M,node=1,billing=49,gres/gpu=13,gres/gpu:3g.20gb=5,gres/gpu:40gb=3,gres/gpu:4g.20gb=5&lt;br /&gt;
   TRESBillingWeights=cpu=1.0,gres/gpu:3g.20gb=6.0,gres/gpu:4g.20gb=6.0,gres/gpu:40gb=6.0,mem=0.05g&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In the example, we have highlighted with  &amp;#039;&amp;#039;&amp;#039;=&amp;gt;&amp;#039;&amp;#039;&amp;#039;  the most relevant default values for Mufasa users, i.e.:&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefaultTime&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default execution time assigned to a job run on the partition (e.g., 1 hour)&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefMemPerNode&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default amount of RAM assigned to a job run on the partition (e.g., 4GB)&lt;br /&gt;
&lt;br /&gt;
= System resources subjected to limitations =&lt;br /&gt;
&lt;br /&gt;
In systems based on SLURM like Mufasa, &amp;#039;&amp;#039;&amp;#039;TRES (Trackable RESources)&amp;#039;&amp;#039;&amp;#039; are (from [https://slurm.schedmd.com/tres.html SLURM&amp;#039;s documentation] &amp;quot;&amp;#039;&amp;#039;resources that can be tracked for usage or used to enforce limits against.&amp;#039;&amp;#039;&amp;quot;&lt;br /&gt;
&lt;br /&gt;
TRES include CPUs, RAM and &amp;#039;&amp;#039;&amp;#039;GRES&amp;#039;&amp;#039;&amp;#039;. The last term stands for &amp;#039;&amp;#039;Generic RESources&amp;#039;&amp;#039; that a job may need for its execution. In Mufasa, the only &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; resources are the GPUs.&lt;br /&gt;
&lt;br /&gt;
== &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax ==&lt;br /&gt;
&lt;br /&gt;
To ask SLURM to assign GRES resources (i.e., GPUs) to a job, a special syntax must be used. Precisely, the name of each GPU resource takes the form&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:Name:Type&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
Considering the [[System#CPUs and GPUs|GPU complement of Mufasa]], &amp;lt;code&amp;gt;Type&amp;lt;/code&amp;gt; takes the following values:&lt;br /&gt;
&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:40gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 40 Gbytes of RAM&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:4g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 4 compute units&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 3 compute units&lt;br /&gt;
&lt;br /&gt;
So, for instance,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
identifies a resource corresponding to a GPU with 20 GB of RAM and 3 compute units.&lt;br /&gt;
&lt;br /&gt;
When asking for a GRES resource (e.g., in an &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command or an &amp;lt;code&amp;gt;SBATCH&amp;lt;/code&amp;gt; directive of an [[User Jobs#Using execution scripts to run jobs|execution script]]), the syntax required by SLURM is&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:&amp;lt;Type&amp;gt;:&amp;lt;Quantity&amp;gt;&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
where &amp;lt;code&amp;gt;Quantity&amp;lt;/code&amp;gt; is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of type &amp;lt;code&amp;gt;4g.20gb&amp;lt;/code&amp;gt; the syntax is&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:4g.20gb:2&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SLURM&amp;#039;s &amp;#039;&amp;#039;generic resources&amp;#039;&amp;#039; are defined in &amp;lt;code&amp;gt;/etc/slurm/gres.conf&amp;lt;/code&amp;gt;. In order to make GPUs available to SLURM&amp;#039;s &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; management, Mufasa makes use of Nvidia&amp;#039;s [https://developer.nvidia.com/nvidia-management-library-nvml NVML library]. For additional information see [https://slurm.schedmd.com/gres.html SLURM&amp;#039;s documentation].&lt;/div&gt;</summary>
		<author><name>GiulioFontana</name></author>
	</entry>
	<entry>
		<id>https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2568</id>
		<title>SLURM</title>
		<link rel="alternate" type="text/html" href="https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2568"/>
		<updated>2026-11-03T14:29:45Z</updated>

		<summary type="html">&lt;p&gt;GiulioFontana: /* Elements determining job priority */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page presents the features of SLURM that are most relevant to Mufasa&amp;#039;s [[Roles|Job Users]]. Job Users can submit jobs for execution, cancel their own jobs, and see other users&amp;#039; jobs (but not intervene on them).&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa &amp;#039;&amp;#039;&amp;#039;must use SLURM&amp;#039;&amp;#039;&amp;#039; to run resource-heavy processes, i.e. computing jobs that require one or more of the following:&lt;br /&gt;
* GPUs&lt;br /&gt;
* multiple CPUs&lt;br /&gt;
* powerful CPUs&lt;br /&gt;
* a significant amount of RAM&lt;br /&gt;
&lt;br /&gt;
In fact, only processes run via SLURM have access to all the resources of Mufasa. Processes run outside SLURM are executed by the [[System#Login server|login server]] virtual machine, which has minimal resources and no GPUs. Using SLURM is therefore the only way to execute resource-heavy jobs on Mufasa (this is a key difference between Mufasa 1.0 and Mufasa 2.0).&lt;br /&gt;
&lt;br /&gt;
= SLURM in a nutshell =&lt;br /&gt;
&lt;br /&gt;
Computation jobs on Mufasa needs to be launched via [[System#The SLURM job scheduling system|SLURM]]. SLURM provides jobs with access to the [[#System resources subjected to limitations|physical resources]] of Mufasa, such as CPUs, GPUs and RAM. Thanks to SLURM, processing jobs share system resources, optimising their occupation and availability. &lt;br /&gt;
&lt;br /&gt;
When a user runs a job, the job does not get executed immediately and is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039;. SLURM executes jobs according to their order in the queue: the top job in the queue gets executed as soon as the necessary resources are available, while jobs lower in the queue wait longer. The position of a job in the queue is due to the &amp;#039;&amp;#039;&amp;#039;[[#Job priority|priority]]&amp;#039;&amp;#039;&amp;#039; assigned to it by SLURM, with higher-priority jobs closer to the top. As a general rule:&lt;br /&gt;
&lt;br /&gt;
;: &amp;#039;&amp;#039;&amp;#039;the greater the fraction of Mufasa&amp;#039;s overall resources that a job asks for, the lower the job&amp;#039;s priority will be&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
The priority mechanism is used to encourage users to use Mufasa&amp;#039;s resources (i.e.: GPUs, CPUs, RAM, execution time) in an effective and equitable manner. This page includes a [[#How_to_maximise_the_priority_of_your_jobs|chart explaining how to maximise the priority of your jobs]].&lt;br /&gt;
&lt;br /&gt;
The &amp;#039;&amp;#039;&amp;#039;time&amp;#039;&amp;#039;&amp;#039; available to a job for its execution is controlled by SLURM. When a user requests execution of a job, they must specify the duration of the time slot that the job needs. The job must complete its execution before the end of the requested time slot, because at the end of such slot it gets killed by SLURM.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0 access to system resources is managed via SLURM&amp;#039;s [[#SLURM Quality of Service (QOS)|Quality of Service (QOS)]] mechanism (Mufasa 1.0 used [[#SLURM_partitions|partitions]] instead). While [[User Jobs#Running jobs with SLURM|launching a processing job via SLURM]], the user must always specify the chosen QOS. QOSes differ in the set of resources that they provide access to because each of them is designed to fit a given type of job.&lt;br /&gt;
&lt;br /&gt;
Mufasa sets limits to the number of jobs by the same user. This page includes a [[#Limits on jobs by the same user|table summarising such limits]].&lt;br /&gt;
&lt;br /&gt;
= SLURM Quality of Service (QOS) =&lt;br /&gt;
&lt;br /&gt;
Through &amp;#039;&amp;#039;&amp;#039;Quality of Services&amp;#039;&amp;#039;&amp;#039; (&amp;#039;&amp;#039;&amp;#039;QOSes&amp;#039;&amp;#039;&amp;#039;), SLURM lets system configurators assign a name to a set of related constraints.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, QOSes are used to define different levels of access to the server&amp;#039;s resources. When [[User Jobs|executing a job with SLURM]], a user must always &amp;#039;&amp;#039;&amp;#039;specify the QOS&amp;#039;&amp;#039;&amp;#039; that their job will use: this choice, in turn, determines what resources the job is able to access and influences the [[#Job priority|priority]] of the job.&lt;br /&gt;
&lt;br /&gt;
Mufasa&amp;#039;s QOSes and their features can be inspected with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list qos format=name%-11,priority,MaxSubmit,maxwall,maxtres%-80,mintres%-18&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following (see below for the same data in table form):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
Name          Priority MaxSubmit     MaxWall MaxTRES                                                                          MinTRES            &lt;br /&gt;
----------- ---------- --------- ----------- -------------------------------------------------------------------------------- ------------------ &lt;br /&gt;
normal               0                                                                                                                           &lt;br /&gt;
nogpu                4         1  3-00:00:00 cpu=16,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=128G                    &lt;br /&gt;
gpuheavy-20          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=2,mem=128G             gres/gpu:4g.20gb=1 &lt;br /&gt;
gpuheavy-40          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=1,gres/gpu:4g.20gb=0,mem=128G             gres/gpu:40gb=1    &lt;br /&gt;
gpulight             8         1    12:00:00 cpu=2,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpu                  2         1  1-00:00:00 cpu=8,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpuwide              2         2  1-00:00:00 cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=1,mem=64G              gres/gpu:4g.20gb=1 &lt;br /&gt;
build               32         1    02:00:00 cpu=2,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=16G&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The columns of this output are the following:&lt;br /&gt;
&lt;br /&gt;
:; Name&lt;br /&gt;
:: name of the QOS&lt;br /&gt;
&lt;br /&gt;
:; Priority&lt;br /&gt;
:: priority tier associated to the QOS (higher value = higher priority): see [[#Job priority|Job priority]] for details&lt;br /&gt;
&lt;br /&gt;
:; MaxSubmit&lt;br /&gt;
:: maximum number of jobs from a single user that can be submitted to SLURM with this QOS; submitted jobs include both running and queued jobs&lt;br /&gt;
:: See [[#Limits on jobs by the same user|Limits on jobs by the same user]] for an overview of the limits on jobs set by Mufasa.&lt;br /&gt;
&lt;br /&gt;
:; MaxWall&lt;br /&gt;
:: maximum wall clock duration of the jobs using the QOS (after which they are killed by SLURM), in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
:: For some QOSes these are not set: it means that they are determined by the [[#SLURM partitions|partition]]. Partitions also define the [[#Default values|default duration]] of jobs.&lt;br /&gt;
&lt;br /&gt;
:; MaxTRES&lt;br /&gt;
:: amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) available to a job using the QOS, where&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;cpu=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of CPUs (i.e., processor cores) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of CPUs specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;gres/&amp;#039;&amp;#039;gpu:Type&amp;#039;&amp;#039;=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of GPUs of class &amp;lt;code&amp;gt;&amp;#039;&amp;#039;Type&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; (see [[User Jobs#gres syntax|&amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax]]) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; (for QOSes that allow access to GPUs) if not specified, the job cannot be launched&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;mem=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;G&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum amount of system RAM is &amp;#039;&amp;#039;K&amp;#039;&amp;#039; GBytes&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of RAM specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
&lt;br /&gt;
:; MinTRES&lt;br /&gt;
:: minimum amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) that a job using the QOS must request in order to actually get executed by SLURM.&lt;br /&gt;
:: (If your job does not actually need these resources, you&amp;#039;ve chosen the wrong QOS: you can use one with a higher priority.)&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; QOS is the default one, and exists only to ensure that users always specify a QOS when running a job. Since &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; has zero priority and no resources, a job run using this QOS would never be run.&lt;br /&gt;
&lt;br /&gt;
The information provided by the &amp;lt;code&amp;gt;sacctmgr list qos&amp;lt;/code&amp;gt; command above is summarised by the following table:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! QOS&lt;br /&gt;
! Priority&amp;lt;/br&amp;gt;tier&lt;br /&gt;
! Max&amp;lt;/br&amp;gt;Submit&lt;br /&gt;
! MaxWall&amp;lt;/br&amp;gt;[h]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;CPUs&lt;br /&gt;
! max RAM&amp;lt;/br&amp;gt;[GB]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;3g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;4g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;40GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | build&lt;br /&gt;
| 32&lt;br /&gt;
| 1&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 16&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpulight&lt;br /&gt;
| 8&lt;br /&gt;
| 1&lt;br /&gt;
| 12&lt;br /&gt;
| 2&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | nogpu&lt;br /&gt;
| 4&lt;br /&gt;
| 1&lt;br /&gt;
| 72&lt;br /&gt;
| 16&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpu&lt;br /&gt;
| 2&lt;br /&gt;
| 1&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuwide&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-20&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| 2 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-40&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
A key piece of information in the table above is the &amp;#039;&amp;#039;&amp;#039;priority tier&amp;#039;&amp;#039;&amp;#039; associated to each QOS. In Mufasa 2.0, priority tiers are used to encourage users to use the &amp;#039;&amp;#039;&amp;#039;least powerful QOS that is compatible with their needs&amp;#039;&amp;#039;&amp;#039;, where &amp;quot;powerful&amp;quot; means &amp;quot;rich with resources&amp;quot;. Less powerful QOSes increase the priority of the jobs that use them, so these jobs tend to be executed sooner. See [[#Job priority|Job priority]] for details about how priority affects the execution order of jobs in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
== The &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS ==&lt;br /&gt;
&lt;br /&gt;
This QOS is specifically designed to be used by Mufasa users to &amp;#039;&amp;#039;&amp;#039;quickly build [[System#Containers|container images]]&amp;#039;&amp;#039;&amp;#039;. Its associated priority tier is very high: this, combined with the fact that jobs using this QOS require few resources, means that such jobs usually get executed very soon.&lt;br /&gt;
&lt;br /&gt;
On the other hand, the limited resources and the lack of access to the GPUs make the &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS unsuitable for other tasks.&lt;br /&gt;
&lt;br /&gt;
See [[Singularity#Building Singularity images|Building Singularity images]] for directions about building Singularity container images.&lt;br /&gt;
&lt;br /&gt;
== Restricted QOSes ==&lt;br /&gt;
&lt;br /&gt;
In Mufasa, the most powerful QOSes are reserved to researchers (called &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users: these include academic personnel and Ph.D. students). M.Sc. students, called &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users, cannot use them while running jobs.&lt;br /&gt;
&lt;br /&gt;
See [[#research users and students users|below]] to understand how &amp;#039;&amp;#039;user categories&amp;#039;&amp;#039; work in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
= &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users =&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa belong to two &amp;#039;&amp;#039;&amp;#039;user categories&amp;#039;&amp;#039;&amp;#039;. Categories provide users with different access to  to Mufasa&amp;#039;s resources: the idea being to provide researchers with more access while still letting students benefit from the server&amp;#039;s capabilities.&lt;br /&gt;
&lt;br /&gt;
User categories in Mufasa are the following:&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. academic personnel and Ph.D. students&lt;br /&gt;
::: * can use all QOSes, including the the [[#Restricted QOSes|restricted ones]]&lt;br /&gt;
::: * their jobs have a higher &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a higher number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. M.Sc. students&lt;br /&gt;
::: * cannot use the [[#Restricted QOSes|restricted QOSes]]&lt;br /&gt;
::: * their jobs have a lower &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a lower number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
You can inspect the differences in priority and running jobs between &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association format=&amp;quot;account,priority,maxjobs&amp;quot; | grep -E &amp;#039;Account|research|students&amp;#039;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
   Account   Priority MaxJobs &lt;br /&gt;
  research          4       2 &lt;br /&gt;
  students          1       1&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
To know what limits apply to your own user, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association where user=$USER format=&amp;quot;user,priority,maxjobs,qos%-60&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
      User   Priority MaxJobs QOS                                                          &lt;br /&gt;
---------- ---------- ------- ------------------------------------------------------------ &lt;br /&gt;
    preali          4       2 build,gpu,gpuheavy-20,gpuheavy-40,gpulight,gpuwide,nogpu&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The list under &amp;quot;QOS&amp;quot; shows what QOSes your user is allowed to use when [[User Jobs|running jobs]]. &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users can use all of them, while &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users can only access a subset of them.&lt;br /&gt;
&lt;br /&gt;
= Limits on jobs by the same user =&lt;br /&gt;
&lt;br /&gt;
Mufasa enforces limits on the number of jobs from a single user. Such limits aim at preventing users from &amp;quot;hogging&amp;quot; system resources, and apply to:&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;submitted jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that the user asked SLURM to execute, each of which may currently be either running or queued&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;running jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that are currently in execution&lt;br /&gt;
&lt;br /&gt;
Limits on submitted jobs are set via [[#SLURM Quality of Service (QOS)|QOSes]], while limits on running jobs are set via [[#research users and students users|user category]].&lt;br /&gt;
&lt;br /&gt;
The following table summarises what limits exist:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! &lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;running&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;submitted&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | global limits&amp;lt;br/&amp;gt;(system-wide)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to limits set by individual QOSes (below)&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | limits for&amp;lt;br/&amp;gt;individual&amp;lt;/br&amp;gt;QOSes&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to global limits (above)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for the&amp;#039;&amp;#039; &amp;lt;code&amp;gt;gpuwide&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;QOS&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for each of the other QOSes&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
= Job priority =&lt;br /&gt;
&lt;br /&gt;
Once the execution of a job has been requested, the job is not run immediately: it is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039; by SLURM, together with all the other jobs awaiting execution. The job on top of the queue at any time is the first to be put into execution as soon as the resources it requires are available. &lt;br /&gt;
&lt;br /&gt;
The order of the items in the job queue depends on their &amp;#039;&amp;#039;&amp;#039;priority&amp;#039;&amp;#039;&amp;#039;. The job with the highest priority is put by SLURM on top of the queue, while all other jobs are put in the queue in order of descending priority.&lt;br /&gt;
&lt;br /&gt;
The goal of SLURM is to maximise resource availability: i.e., to ensure the shorter possible wait time before job execution. To achieve this goal, in Mufasa SLURM is configured to &amp;#039;&amp;#039;&amp;#039;encourage users not to ask for resources or execution time that their job doesn&amp;#039;t need&amp;#039;&amp;#039;&amp;#039;. This is done via the priority mechanism: the more resources and/or time a job requests, the lower its priority will be; and the later it will be executed.&lt;br /&gt;
&lt;br /&gt;
Priority management in Mufasa is designed to set up a &amp;#039;&amp;#039;&amp;#039;virtuous cycle&amp;#039;&amp;#039;&amp;#039; where users, by carefully choosing what to ask for, obtain two results:&lt;br /&gt;
* they ensure that their job is executed as soon as possible;&lt;br /&gt;
* they leave as much as possible of Mufasa&amp;#039;s resources free for other users&amp;#039;s jobs.&lt;br /&gt;
&lt;br /&gt;
== Elements determining job priority ==&lt;br /&gt;
In Mufasa, the priority of a job is computed by SLURM according to the following elements:&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#research users and students users|User category]]&amp;#039;&amp;#039;&amp;#039; (i.e., &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;)&lt;br /&gt;
::: Used to provide higher priority to jobs run by &amp;#039;&amp;#039;&amp;#039;research personnel&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#SLURM Quality of Service (QOS)|QOS]]&amp;#039;&amp;#039;&amp;#039; used by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;less resources&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Number of CPUs&amp;#039;&amp;#039;&amp;#039; requested by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;a lower number of CPUs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job Age&amp;#039;&amp;#039;&amp;#039;, i.e. the time that the job has been waiting in the queue&lt;br /&gt;
::: Used to provide higher priority to jobs which have been &amp;#039;&amp;#039;&amp;#039;queued for a longer time&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;FairShare&amp;#039;&amp;#039;&amp;#039;, i.e. a factor computed by SLURM to balance use of the system by different users&lt;br /&gt;
::: Used to provide higher priority to jobs by users who &amp;#039;&amp;#039;&amp;#039;used Mufasa&amp;#039;s resources less than others&amp;#039;&amp;#039;&amp;#039; (see [[#How FairShare works|below]])&lt;br /&gt;
&lt;br /&gt;
You can understand how each of these elements contribute to the priority of your jobs using command &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;. A tailored version of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; that only displays the parameters listed above, in the same order, is the following:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sprio -o &amp;quot;%6i %9u %.5Y %.10B %.5Q %.9T %.5A %.5F&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
Here is an example of its output:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
JOBID  USER      PRIOR      ASSOC   QOS      TRES   AGE FAIRS&lt;br /&gt;
5774   itombini   3492       2500   750  cpu=-800   906   136&lt;br /&gt;
5780   fbottazzi  2266        625   750  cpu=-600   420  1071&lt;br /&gt;
5783   mzoccola   1766        625   750  cpu=-800   217   974&lt;br /&gt;
5790   amoglia    2717       2500   375  cpu=-800   156   487&lt;br /&gt;
5791   qliu       2232       2500   375  cpu=-800   138    19&lt;br /&gt;
5792   fpastori   2197       2500   375  cpu=-800    84    39&lt;br /&gt;
5793   bagostini  2034        625  1500 cpu=-1200    18  1091&lt;br /&gt;
5794   lpassoni   2517       2500   750  cpu=-800     9    58&lt;br /&gt;
5795   lpassoni   2517       2500   750  cpu=-800     9    58&lt;br /&gt;
5796   adurat     3101       2500   750  cpu=-800     8   643&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
In it, column ASSOC (i.e., association) corresponds to user category, while column TRES corresponds to the number of CPUs. Note that the contribution of CPUs to priority is &amp;#039;&amp;#039;negative&amp;#039;&amp;#039;, i.e. it is a penalty increasing with the number of CPUs requested by the job. Column PRIOR provides the overall priority of each job.&lt;br /&gt;
&lt;br /&gt;
=== How FairShare works ===&lt;br /&gt;
In Mufasa 2.0, the FairShare of each user is computed according to these rules:&lt;br /&gt;
# It considers the quantity used by (current and past) user jobs of:&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;CPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of CPUs used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;RAM&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the amount of RAM used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;GPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of GPUs used&lt;br /&gt;
#** note: 40 GB GPUs have double the impact on FairShare than 20 GB GPUs&lt;br /&gt;
# The impact on FairShare of any use of resources is &amp;#039;&amp;#039;&amp;#039;proportional to its duration&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
#* example: using 32 GB of RAM for 48h has the same impact of using 64 GB of RAM for 24h&lt;br /&gt;
# FairShare has a &amp;quot;fading memory&amp;quot;: i.e., resource use has less and less impact on FairShare the farther in the past it is&lt;br /&gt;
#* consequence: over time, the &amp;quot;history&amp;quot; of a user gets forgotten by FairShare&lt;br /&gt;
&lt;br /&gt;
== How to maximise the priority of your jobs ==&lt;br /&gt;
&lt;br /&gt;
Every time you run a SLURM job, follow these guidelines:&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Choose the less powerful QOS compatible with the needs of your job&lt;br /&gt;
:: QOSes with access to less resources have higher base priority&lt;br /&gt;
&lt;br /&gt;
; Only request CPUs and RAM that your job really needs&lt;br /&gt;
:: Usually, code exploits multiple CPUs only if specifically designed to do so: if unsure, only ask for 1 CPU&lt;br /&gt;
:: The fewer resources your job asks for, the less it will wait before they become available&lt;br /&gt;
:: Asking for fewer CPUs and/or less RAM improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Do not request more time than your jobs needs to complete&lt;br /&gt;
:: Make a worst-case estimate and only ask for that duration&lt;br /&gt;
:: Asking for less of Mufasa&amp;#039;s time improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Debug and test code using less powerful QOSes before running it with more powerful QOSes&lt;br /&gt;
:: Your test jobs will get a higher priority and your FairShare will improve&lt;br /&gt;
&lt;br /&gt;
; Cancel jobs that you don&amp;#039;t need them anymore&lt;br /&gt;
:: Always use [[User_Jobs#Cancelling_a_job_with_scancel|scancel]] to delete completed (or crashed) jobs: your Fairshare will improve&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Protip: it&amp;#039;s a good idea to [[User Jobs#Looking for unused GPUs|check for unused GPUs]] before choosing what to request. Requesting a GPU that is currently idle will help your job get executed sooner.&lt;br /&gt;
&lt;br /&gt;
= SLURM partitions =&lt;br /&gt;
&lt;br /&gt;
Partitions are another mechanism provided by SLURM to create different levels of access to system resources. Since in Mufasa 2.0 access to resources is controlled via [[#SLURM Quality of Service (QOS)|QOSes]], partitions are not very relevant. (This is one of the main differences between Mufasa 1.0 and Mufasa 2.0.)&lt;br /&gt;
&lt;br /&gt;
Note, however, that the default values for some features of SLURM jobs (e.g., duration) are [[#Default values|set by the partition]].&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, there is a single SLURM partition, called &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt;, and all jobs run on it. The state of &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt; can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sinfo -o &amp;quot;%10P %5a %9T %11L %10l&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PARTITION  AVAIL STATE     DEFAULTTIME TIMELIMIT &lt;br /&gt;
jobs*      up    idle      1:00:00     3-00:00:00&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where columns correspond to the following information:&lt;br /&gt;
&lt;br /&gt;
:; PARTITION&lt;br /&gt;
:: name of the partition; the asterisks indicates that it&amp;#039;s the default one&lt;br /&gt;
&lt;br /&gt;
:; AVAIL&lt;br /&gt;
:: state/availability of the partition: see [[#Partition availability|below]]&lt;br /&gt;
&lt;br /&gt;
:; STATE&lt;br /&gt;
:: state (using [https://slurm.schedmd.com/sinfo.html#SECTION_NODE-STATE-CODES these codes])&lt;br /&gt;
:: typical values are &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;mixed&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that some of the resources are busy executing jobs while other are idle, and &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;allocated&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that all of the resources are in use&lt;br /&gt;
&lt;br /&gt;
:; DEFAULTTIME&lt;br /&gt;
:: default runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:; TIMELIMIT&lt;br /&gt;
:: maximum runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
The asterisk at the end of the partition name indicates the default partition, i.e. the one on which jobs which do not ask for a specific partition are run.&lt;br /&gt;
&lt;br /&gt;
Command &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; does not tell you about the &amp;#039;&amp;#039;jobs&amp;#039;&amp;#039; submitted to a partition. This information is obtained, instead, with [[User Jobs#Inspecting jobs with squeue|command &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;]].&lt;br /&gt;
&lt;br /&gt;
== Partition availability ==&lt;br /&gt;
&lt;br /&gt;
The most important information that &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; provides is the &amp;#039;&amp;#039;&amp;#039;availability&amp;#039;&amp;#039;&amp;#039; (also called &amp;#039;&amp;#039;&amp;#039;state&amp;#039;&amp;#039;&amp;#039;) of partitions. This is shown in column &amp;quot;AVAIL&amp;quot;. Possible partition states are:&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is available&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed as soon as resources allow&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is in the process of becoming unavailable (i.e., of entering the &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; state: see below)&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is unavailable&lt;br /&gt;
:: There are no running jobs&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
When a partition goes from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; no harm is done to running jobs. In a normally functioning SLURM system, the passage from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; happens only when no jobs are running on the partition. If (e.g., due to a malfunction) the passage happens with jobs still running, they get killed.&lt;br /&gt;
&lt;br /&gt;
A partition in state &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; requires intervention by a [[Roles|Job Administrator]] to be restored to &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
== Default values ==&lt;br /&gt;
&lt;br /&gt;
The features of SLURM partitions, including the &amp;#039;&amp;#039;&amp;#039;default values&amp;#039;&amp;#039;&amp;#039; which are applied to jobs that do not make explicit requests, can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scontrol show partition&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PartitionName=jobs&lt;br /&gt;
   AllowGroups=ALL AllowAccounts=ALL AllowQos=nogpu,gpulight,gpu,gpuwide,gpuheavy-20,gpuheavy-40&lt;br /&gt;
   AllocNodes=ALL Default=YES QoS=N/A&lt;br /&gt;
=&amp;gt; DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO&lt;br /&gt;
   MaxNodes=UNLIMITED MaxTime=3-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED&lt;br /&gt;
   Nodes=gn01&lt;br /&gt;
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO&lt;br /&gt;
   OverTimeLimit=NONE PreemptMode=OFF&lt;br /&gt;
   State=UP TotalCPUs=48 TotalNodes=1 SelectTypeParameters=NONE&lt;br /&gt;
   JobDefaults=(null)&lt;br /&gt;
=&amp;gt; DefMemPerNode=4096 MaxMemPerNode=UNLIMITED&lt;br /&gt;
   TRES=cpu=48,mem=1011435M,node=1,billing=49,gres/gpu=13,gres/gpu:3g.20gb=5,gres/gpu:40gb=3,gres/gpu:4g.20gb=5&lt;br /&gt;
   TRESBillingWeights=cpu=1.0,gres/gpu:3g.20gb=6.0,gres/gpu:4g.20gb=6.0,gres/gpu:40gb=6.0,mem=0.05g&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In the example, we have highlighted with  &amp;#039;&amp;#039;&amp;#039;=&amp;gt;&amp;#039;&amp;#039;&amp;#039;  the most relevant default values for Mufasa users, i.e.:&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefaultTime&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default execution time assigned to a job run on the partition (e.g., 1 hour)&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefMemPerNode&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default amount of RAM assigned to a job run on the partition (e.g., 4GB)&lt;br /&gt;
&lt;br /&gt;
= System resources subjected to limitations =&lt;br /&gt;
&lt;br /&gt;
In systems based on SLURM like Mufasa, &amp;#039;&amp;#039;&amp;#039;TRES (Trackable RESources)&amp;#039;&amp;#039;&amp;#039; are (from [https://slurm.schedmd.com/tres.html SLURM&amp;#039;s documentation] &amp;quot;&amp;#039;&amp;#039;resources that can be tracked for usage or used to enforce limits against.&amp;#039;&amp;#039;&amp;quot;&lt;br /&gt;
&lt;br /&gt;
TRES include CPUs, RAM and &amp;#039;&amp;#039;&amp;#039;GRES&amp;#039;&amp;#039;&amp;#039;. The last term stands for &amp;#039;&amp;#039;Generic RESources&amp;#039;&amp;#039; that a job may need for its execution. In Mufasa, the only &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; resources are the GPUs.&lt;br /&gt;
&lt;br /&gt;
== &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax ==&lt;br /&gt;
&lt;br /&gt;
To ask SLURM to assign GRES resources (i.e., GPUs) to a job, a special syntax must be used. Precisely, the name of each GPU resource takes the form&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:Name:Type&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
Considering the [[System#CPUs and GPUs|GPU complement of Mufasa]], &amp;lt;code&amp;gt;Type&amp;lt;/code&amp;gt; takes the following values:&lt;br /&gt;
&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:40gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 40 Gbytes of RAM&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:4g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 4 compute units&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 3 compute units&lt;br /&gt;
&lt;br /&gt;
So, for instance,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
identifies a resource corresponding to a GPU with 20 GB of RAM and 3 compute units.&lt;br /&gt;
&lt;br /&gt;
When asking for a GRES resource (e.g., in an &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command or an &amp;lt;code&amp;gt;SBATCH&amp;lt;/code&amp;gt; directive of an [[User Jobs#Using execution scripts to run jobs|execution script]]), the syntax required by SLURM is&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:&amp;lt;Type&amp;gt;:&amp;lt;Quantity&amp;gt;&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
where &amp;lt;code&amp;gt;Quantity&amp;lt;/code&amp;gt; is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of type &amp;lt;code&amp;gt;4g.20gb&amp;lt;/code&amp;gt; the syntax is&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:4g.20gb:2&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SLURM&amp;#039;s &amp;#039;&amp;#039;generic resources&amp;#039;&amp;#039; are defined in &amp;lt;code&amp;gt;/etc/slurm/gres.conf&amp;lt;/code&amp;gt;. In order to make GPUs available to SLURM&amp;#039;s &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; management, Mufasa makes use of Nvidia&amp;#039;s [https://developer.nvidia.com/nvidia-management-library-nvml NVML library]. For additional information see [https://slurm.schedmd.com/gres.html SLURM&amp;#039;s documentation].&lt;/div&gt;</summary>
		<author><name>GiulioFontana</name></author>
	</entry>
	<entry>
		<id>https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2567</id>
		<title>SLURM</title>
		<link rel="alternate" type="text/html" href="https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2567"/>
		<updated>2026-11-03T14:29:16Z</updated>

		<summary type="html">&lt;p&gt;GiulioFontana: /* Elements determining job priority */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page presents the features of SLURM that are most relevant to Mufasa&amp;#039;s [[Roles|Job Users]]. Job Users can submit jobs for execution, cancel their own jobs, and see other users&amp;#039; jobs (but not intervene on them).&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa &amp;#039;&amp;#039;&amp;#039;must use SLURM&amp;#039;&amp;#039;&amp;#039; to run resource-heavy processes, i.e. computing jobs that require one or more of the following:&lt;br /&gt;
* GPUs&lt;br /&gt;
* multiple CPUs&lt;br /&gt;
* powerful CPUs&lt;br /&gt;
* a significant amount of RAM&lt;br /&gt;
&lt;br /&gt;
In fact, only processes run via SLURM have access to all the resources of Mufasa. Processes run outside SLURM are executed by the [[System#Login server|login server]] virtual machine, which has minimal resources and no GPUs. Using SLURM is therefore the only way to execute resource-heavy jobs on Mufasa (this is a key difference between Mufasa 1.0 and Mufasa 2.0).&lt;br /&gt;
&lt;br /&gt;
= SLURM in a nutshell =&lt;br /&gt;
&lt;br /&gt;
Computation jobs on Mufasa needs to be launched via [[System#The SLURM job scheduling system|SLURM]]. SLURM provides jobs with access to the [[#System resources subjected to limitations|physical resources]] of Mufasa, such as CPUs, GPUs and RAM. Thanks to SLURM, processing jobs share system resources, optimising their occupation and availability. &lt;br /&gt;
&lt;br /&gt;
When a user runs a job, the job does not get executed immediately and is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039;. SLURM executes jobs according to their order in the queue: the top job in the queue gets executed as soon as the necessary resources are available, while jobs lower in the queue wait longer. The position of a job in the queue is due to the &amp;#039;&amp;#039;&amp;#039;[[#Job priority|priority]]&amp;#039;&amp;#039;&amp;#039; assigned to it by SLURM, with higher-priority jobs closer to the top. As a general rule:&lt;br /&gt;
&lt;br /&gt;
;: &amp;#039;&amp;#039;&amp;#039;the greater the fraction of Mufasa&amp;#039;s overall resources that a job asks for, the lower the job&amp;#039;s priority will be&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
The priority mechanism is used to encourage users to use Mufasa&amp;#039;s resources (i.e.: GPUs, CPUs, RAM, execution time) in an effective and equitable manner. This page includes a [[#How_to_maximise_the_priority_of_your_jobs|chart explaining how to maximise the priority of your jobs]].&lt;br /&gt;
&lt;br /&gt;
The &amp;#039;&amp;#039;&amp;#039;time&amp;#039;&amp;#039;&amp;#039; available to a job for its execution is controlled by SLURM. When a user requests execution of a job, they must specify the duration of the time slot that the job needs. The job must complete its execution before the end of the requested time slot, because at the end of such slot it gets killed by SLURM.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0 access to system resources is managed via SLURM&amp;#039;s [[#SLURM Quality of Service (QOS)|Quality of Service (QOS)]] mechanism (Mufasa 1.0 used [[#SLURM_partitions|partitions]] instead). While [[User Jobs#Running jobs with SLURM|launching a processing job via SLURM]], the user must always specify the chosen QOS. QOSes differ in the set of resources that they provide access to because each of them is designed to fit a given type of job.&lt;br /&gt;
&lt;br /&gt;
Mufasa sets limits to the number of jobs by the same user. This page includes a [[#Limits on jobs by the same user|table summarising such limits]].&lt;br /&gt;
&lt;br /&gt;
= SLURM Quality of Service (QOS) =&lt;br /&gt;
&lt;br /&gt;
Through &amp;#039;&amp;#039;&amp;#039;Quality of Services&amp;#039;&amp;#039;&amp;#039; (&amp;#039;&amp;#039;&amp;#039;QOSes&amp;#039;&amp;#039;&amp;#039;), SLURM lets system configurators assign a name to a set of related constraints.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, QOSes are used to define different levels of access to the server&amp;#039;s resources. When [[User Jobs|executing a job with SLURM]], a user must always &amp;#039;&amp;#039;&amp;#039;specify the QOS&amp;#039;&amp;#039;&amp;#039; that their job will use: this choice, in turn, determines what resources the job is able to access and influences the [[#Job priority|priority]] of the job.&lt;br /&gt;
&lt;br /&gt;
Mufasa&amp;#039;s QOSes and their features can be inspected with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list qos format=name%-11,priority,MaxSubmit,maxwall,maxtres%-80,mintres%-18&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following (see below for the same data in table form):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
Name          Priority MaxSubmit     MaxWall MaxTRES                                                                          MinTRES            &lt;br /&gt;
----------- ---------- --------- ----------- -------------------------------------------------------------------------------- ------------------ &lt;br /&gt;
normal               0                                                                                                                           &lt;br /&gt;
nogpu                4         1  3-00:00:00 cpu=16,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=128G                    &lt;br /&gt;
gpuheavy-20          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=2,mem=128G             gres/gpu:4g.20gb=1 &lt;br /&gt;
gpuheavy-40          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=1,gres/gpu:4g.20gb=0,mem=128G             gres/gpu:40gb=1    &lt;br /&gt;
gpulight             8         1    12:00:00 cpu=2,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpu                  2         1  1-00:00:00 cpu=8,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpuwide              2         2  1-00:00:00 cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=1,mem=64G              gres/gpu:4g.20gb=1 &lt;br /&gt;
build               32         1    02:00:00 cpu=2,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=16G&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The columns of this output are the following:&lt;br /&gt;
&lt;br /&gt;
:; Name&lt;br /&gt;
:: name of the QOS&lt;br /&gt;
&lt;br /&gt;
:; Priority&lt;br /&gt;
:: priority tier associated to the QOS (higher value = higher priority): see [[#Job priority|Job priority]] for details&lt;br /&gt;
&lt;br /&gt;
:; MaxSubmit&lt;br /&gt;
:: maximum number of jobs from a single user that can be submitted to SLURM with this QOS; submitted jobs include both running and queued jobs&lt;br /&gt;
:: See [[#Limits on jobs by the same user|Limits on jobs by the same user]] for an overview of the limits on jobs set by Mufasa.&lt;br /&gt;
&lt;br /&gt;
:; MaxWall&lt;br /&gt;
:: maximum wall clock duration of the jobs using the QOS (after which they are killed by SLURM), in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
:: For some QOSes these are not set: it means that they are determined by the [[#SLURM partitions|partition]]. Partitions also define the [[#Default values|default duration]] of jobs.&lt;br /&gt;
&lt;br /&gt;
:; MaxTRES&lt;br /&gt;
:: amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) available to a job using the QOS, where&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;cpu=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of CPUs (i.e., processor cores) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of CPUs specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;gres/&amp;#039;&amp;#039;gpu:Type&amp;#039;&amp;#039;=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of GPUs of class &amp;lt;code&amp;gt;&amp;#039;&amp;#039;Type&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; (see [[User Jobs#gres syntax|&amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax]]) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; (for QOSes that allow access to GPUs) if not specified, the job cannot be launched&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;mem=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;G&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum amount of system RAM is &amp;#039;&amp;#039;K&amp;#039;&amp;#039; GBytes&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of RAM specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
&lt;br /&gt;
:; MinTRES&lt;br /&gt;
:: minimum amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) that a job using the QOS must request in order to actually get executed by SLURM.&lt;br /&gt;
:: (If your job does not actually need these resources, you&amp;#039;ve chosen the wrong QOS: you can use one with a higher priority.)&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; QOS is the default one, and exists only to ensure that users always specify a QOS when running a job. Since &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; has zero priority and no resources, a job run using this QOS would never be run.&lt;br /&gt;
&lt;br /&gt;
The information provided by the &amp;lt;code&amp;gt;sacctmgr list qos&amp;lt;/code&amp;gt; command above is summarised by the following table:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! QOS&lt;br /&gt;
! Priority&amp;lt;/br&amp;gt;tier&lt;br /&gt;
! Max&amp;lt;/br&amp;gt;Submit&lt;br /&gt;
! MaxWall&amp;lt;/br&amp;gt;[h]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;CPUs&lt;br /&gt;
! max RAM&amp;lt;/br&amp;gt;[GB]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;3g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;4g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;40GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | build&lt;br /&gt;
| 32&lt;br /&gt;
| 1&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 16&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpulight&lt;br /&gt;
| 8&lt;br /&gt;
| 1&lt;br /&gt;
| 12&lt;br /&gt;
| 2&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | nogpu&lt;br /&gt;
| 4&lt;br /&gt;
| 1&lt;br /&gt;
| 72&lt;br /&gt;
| 16&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpu&lt;br /&gt;
| 2&lt;br /&gt;
| 1&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuwide&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-20&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| 2 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-40&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
A key piece of information in the table above is the &amp;#039;&amp;#039;&amp;#039;priority tier&amp;#039;&amp;#039;&amp;#039; associated to each QOS. In Mufasa 2.0, priority tiers are used to encourage users to use the &amp;#039;&amp;#039;&amp;#039;least powerful QOS that is compatible with their needs&amp;#039;&amp;#039;&amp;#039;, where &amp;quot;powerful&amp;quot; means &amp;quot;rich with resources&amp;quot;. Less powerful QOSes increase the priority of the jobs that use them, so these jobs tend to be executed sooner. See [[#Job priority|Job priority]] for details about how priority affects the execution order of jobs in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
== The &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS ==&lt;br /&gt;
&lt;br /&gt;
This QOS is specifically designed to be used by Mufasa users to &amp;#039;&amp;#039;&amp;#039;quickly build [[System#Containers|container images]]&amp;#039;&amp;#039;&amp;#039;. Its associated priority tier is very high: this, combined with the fact that jobs using this QOS require few resources, means that such jobs usually get executed very soon.&lt;br /&gt;
&lt;br /&gt;
On the other hand, the limited resources and the lack of access to the GPUs make the &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS unsuitable for other tasks.&lt;br /&gt;
&lt;br /&gt;
See [[Singularity#Building Singularity images|Building Singularity images]] for directions about building Singularity container images.&lt;br /&gt;
&lt;br /&gt;
== Restricted QOSes ==&lt;br /&gt;
&lt;br /&gt;
In Mufasa, the most powerful QOSes are reserved to researchers (called &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users: these include academic personnel and Ph.D. students). M.Sc. students, called &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users, cannot use them while running jobs.&lt;br /&gt;
&lt;br /&gt;
See [[#research users and students users|below]] to understand how &amp;#039;&amp;#039;user categories&amp;#039;&amp;#039; work in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
= &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users =&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa belong to two &amp;#039;&amp;#039;&amp;#039;user categories&amp;#039;&amp;#039;&amp;#039;. Categories provide users with different access to  to Mufasa&amp;#039;s resources: the idea being to provide researchers with more access while still letting students benefit from the server&amp;#039;s capabilities.&lt;br /&gt;
&lt;br /&gt;
User categories in Mufasa are the following:&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. academic personnel and Ph.D. students&lt;br /&gt;
::: * can use all QOSes, including the the [[#Restricted QOSes|restricted ones]]&lt;br /&gt;
::: * their jobs have a higher &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a higher number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. M.Sc. students&lt;br /&gt;
::: * cannot use the [[#Restricted QOSes|restricted QOSes]]&lt;br /&gt;
::: * their jobs have a lower &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a lower number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
You can inspect the differences in priority and running jobs between &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association format=&amp;quot;account,priority,maxjobs&amp;quot; | grep -E &amp;#039;Account|research|students&amp;#039;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
   Account   Priority MaxJobs &lt;br /&gt;
  research          4       2 &lt;br /&gt;
  students          1       1&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
To know what limits apply to your own user, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association where user=$USER format=&amp;quot;user,priority,maxjobs,qos%-60&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
      User   Priority MaxJobs QOS                                                          &lt;br /&gt;
---------- ---------- ------- ------------------------------------------------------------ &lt;br /&gt;
    preali          4       2 build,gpu,gpuheavy-20,gpuheavy-40,gpulight,gpuwide,nogpu&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The list under &amp;quot;QOS&amp;quot; shows what QOSes your user is allowed to use when [[User Jobs|running jobs]]. &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users can use all of them, while &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users can only access a subset of them.&lt;br /&gt;
&lt;br /&gt;
= Limits on jobs by the same user =&lt;br /&gt;
&lt;br /&gt;
Mufasa enforces limits on the number of jobs from a single user. Such limits aim at preventing users from &amp;quot;hogging&amp;quot; system resources, and apply to:&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;submitted jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that the user asked SLURM to execute, each of which may currently be either running or queued&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;running jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that are currently in execution&lt;br /&gt;
&lt;br /&gt;
Limits on submitted jobs are set via [[#SLURM Quality of Service (QOS)|QOSes]], while limits on running jobs are set via [[#research users and students users|user category]].&lt;br /&gt;
&lt;br /&gt;
The following table summarises what limits exist:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! &lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;running&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;submitted&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | global limits&amp;lt;br/&amp;gt;(system-wide)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to limits set by individual QOSes (below)&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | limits for&amp;lt;br/&amp;gt;individual&amp;lt;/br&amp;gt;QOSes&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to global limits (above)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for the&amp;#039;&amp;#039; &amp;lt;code&amp;gt;gpuwide&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;QOS&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for each of the other QOSes&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
= Job priority =&lt;br /&gt;
&lt;br /&gt;
Once the execution of a job has been requested, the job is not run immediately: it is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039; by SLURM, together with all the other jobs awaiting execution. The job on top of the queue at any time is the first to be put into execution as soon as the resources it requires are available. &lt;br /&gt;
&lt;br /&gt;
The order of the items in the job queue depends on their &amp;#039;&amp;#039;&amp;#039;priority&amp;#039;&amp;#039;&amp;#039;. The job with the highest priority is put by SLURM on top of the queue, while all other jobs are put in the queue in order of descending priority.&lt;br /&gt;
&lt;br /&gt;
The goal of SLURM is to maximise resource availability: i.e., to ensure the shorter possible wait time before job execution. To achieve this goal, in Mufasa SLURM is configured to &amp;#039;&amp;#039;&amp;#039;encourage users not to ask for resources or execution time that their job doesn&amp;#039;t need&amp;#039;&amp;#039;&amp;#039;. This is done via the priority mechanism: the more resources and/or time a job requests, the lower its priority will be; and the later it will be executed.&lt;br /&gt;
&lt;br /&gt;
Priority management in Mufasa is designed to set up a &amp;#039;&amp;#039;&amp;#039;virtuous cycle&amp;#039;&amp;#039;&amp;#039; where users, by carefully choosing what to ask for, obtain two results:&lt;br /&gt;
* they ensure that their job is executed as soon as possible;&lt;br /&gt;
* they leave as much as possible of Mufasa&amp;#039;s resources free for other users&amp;#039;s jobs.&lt;br /&gt;
&lt;br /&gt;
== Elements determining job priority ==&lt;br /&gt;
In Mufasa, the priority of a job is computed by SLURM according to the following elements:&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#research users and students users|User category]]&amp;#039;&amp;#039;&amp;#039; (i.e., &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;)&lt;br /&gt;
::: Used to provide higher priority to jobs run by &amp;#039;&amp;#039;&amp;#039;research personnel&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#SLURM Quality of Service (QOS)|QOS]]&amp;#039;&amp;#039;&amp;#039; used by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;less resources&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Number of CPUs&amp;#039;&amp;#039;&amp;#039; requested by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;a lower number of CPUs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job Age&amp;#039;&amp;#039;&amp;#039;, i.e. the time that the job has been waiting in the queue&lt;br /&gt;
::: Used to provide higher priority to jobs which have been &amp;#039;&amp;#039;&amp;#039;queued for a longer time&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;FairShare&amp;#039;&amp;#039;&amp;#039;, i.e. a factor computed by SLURM to balance use of the system by different users&lt;br /&gt;
::: Used to provide higher priority to jobs by users who &amp;#039;&amp;#039;&amp;#039;used Mufasa&amp;#039;s resources less than others&amp;#039;&amp;#039;&amp;#039; (see [[#How FairShare works|below]])&lt;br /&gt;
&lt;br /&gt;
You can understand how each of these elements contribute to the priority of your jobs using command &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;. A tailored version of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; that only displays the parameters listed above, in the same order, is the following:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sprio -o &amp;quot;%6i %9u %.5Y %.10B %.5Q %.9T %.5A %.5F&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
Here is an example of its output:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
JOBID  USER      PRIOR      ASSOC   QOS      TRES   AGE FAIRS&lt;br /&gt;
5774   itombini   3492       2500   750  cpu=-800   906   136&lt;br /&gt;
5780   fbottazzi  2266        625   750  cpu=-600   420  1071&lt;br /&gt;
5783   mzoccola   1766        625   750  cpu=-800   217   974&lt;br /&gt;
5790   amoglia    2717       2500   375  cpu=-800   156   487&lt;br /&gt;
5791   qliu       2232       2500   375  cpu=-800   138    19&lt;br /&gt;
5792   fpastori   2197       2500   375  cpu=-800    84    39&lt;br /&gt;
5793   bagostini  2034        625  1500 cpu=-1200    18  1091&lt;br /&gt;
5794   lpassoni   2517       2500   750  cpu=-800     9    58&lt;br /&gt;
5795   lpassoni   2517       2500   750  cpu=-800     9    58&lt;br /&gt;
5796   adurat     3101       2500   750  cpu=-800     8   643&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
In it, column ASSOC (i.e., association) corresponds to user category, while column TRES corresponds to the number of CPUs. Note that the contribution of CPUs to priority is &amp;#039;negative&amp;#039;, i.e. it is a penalty increasing with the number of CPUs requested by the job. Column PRIOR provides the overall priority of each job.&lt;br /&gt;
&lt;br /&gt;
=== How FairShare works ===&lt;br /&gt;
In Mufasa 2.0, the FairShare of each user is computed according to these rules:&lt;br /&gt;
# It considers the quantity used by (current and past) user jobs of:&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;CPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of CPUs used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;RAM&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the amount of RAM used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;GPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of GPUs used&lt;br /&gt;
#** note: 40 GB GPUs have double the impact on FairShare than 20 GB GPUs&lt;br /&gt;
# The impact on FairShare of any use of resources is &amp;#039;&amp;#039;&amp;#039;proportional to its duration&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
#* example: using 32 GB of RAM for 48h has the same impact of using 64 GB of RAM for 24h&lt;br /&gt;
# FairShare has a &amp;quot;fading memory&amp;quot;: i.e., resource use has less and less impact on FairShare the farther in the past it is&lt;br /&gt;
#* consequence: over time, the &amp;quot;history&amp;quot; of a user gets forgotten by FairShare&lt;br /&gt;
&lt;br /&gt;
== How to maximise the priority of your jobs ==&lt;br /&gt;
&lt;br /&gt;
Every time you run a SLURM job, follow these guidelines:&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Choose the less powerful QOS compatible with the needs of your job&lt;br /&gt;
:: QOSes with access to less resources have higher base priority&lt;br /&gt;
&lt;br /&gt;
; Only request CPUs and RAM that your job really needs&lt;br /&gt;
:: Usually, code exploits multiple CPUs only if specifically designed to do so: if unsure, only ask for 1 CPU&lt;br /&gt;
:: The fewer resources your job asks for, the less it will wait before they become available&lt;br /&gt;
:: Asking for fewer CPUs and/or less RAM improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Do not request more time than your jobs needs to complete&lt;br /&gt;
:: Make a worst-case estimate and only ask for that duration&lt;br /&gt;
:: Asking for less of Mufasa&amp;#039;s time improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Debug and test code using less powerful QOSes before running it with more powerful QOSes&lt;br /&gt;
:: Your test jobs will get a higher priority and your FairShare will improve&lt;br /&gt;
&lt;br /&gt;
; Cancel jobs that you don&amp;#039;t need them anymore&lt;br /&gt;
:: Always use [[User_Jobs#Cancelling_a_job_with_scancel|scancel]] to delete completed (or crashed) jobs: your Fairshare will improve&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Protip: it&amp;#039;s a good idea to [[User Jobs#Looking for unused GPUs|check for unused GPUs]] before choosing what to request. Requesting a GPU that is currently idle will help your job get executed sooner.&lt;br /&gt;
&lt;br /&gt;
= SLURM partitions =&lt;br /&gt;
&lt;br /&gt;
Partitions are another mechanism provided by SLURM to create different levels of access to system resources. Since in Mufasa 2.0 access to resources is controlled via [[#SLURM Quality of Service (QOS)|QOSes]], partitions are not very relevant. (This is one of the main differences between Mufasa 1.0 and Mufasa 2.0.)&lt;br /&gt;
&lt;br /&gt;
Note, however, that the default values for some features of SLURM jobs (e.g., duration) are [[#Default values|set by the partition]].&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, there is a single SLURM partition, called &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt;, and all jobs run on it. The state of &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt; can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sinfo -o &amp;quot;%10P %5a %9T %11L %10l&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PARTITION  AVAIL STATE     DEFAULTTIME TIMELIMIT &lt;br /&gt;
jobs*      up    idle      1:00:00     3-00:00:00&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where columns correspond to the following information:&lt;br /&gt;
&lt;br /&gt;
:; PARTITION&lt;br /&gt;
:: name of the partition; the asterisks indicates that it&amp;#039;s the default one&lt;br /&gt;
&lt;br /&gt;
:; AVAIL&lt;br /&gt;
:: state/availability of the partition: see [[#Partition availability|below]]&lt;br /&gt;
&lt;br /&gt;
:; STATE&lt;br /&gt;
:: state (using [https://slurm.schedmd.com/sinfo.html#SECTION_NODE-STATE-CODES these codes])&lt;br /&gt;
:: typical values are &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;mixed&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that some of the resources are busy executing jobs while other are idle, and &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;allocated&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that all of the resources are in use&lt;br /&gt;
&lt;br /&gt;
:; DEFAULTTIME&lt;br /&gt;
:: default runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:; TIMELIMIT&lt;br /&gt;
:: maximum runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
The asterisk at the end of the partition name indicates the default partition, i.e. the one on which jobs which do not ask for a specific partition are run.&lt;br /&gt;
&lt;br /&gt;
Command &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; does not tell you about the &amp;#039;&amp;#039;jobs&amp;#039;&amp;#039; submitted to a partition. This information is obtained, instead, with [[User Jobs#Inspecting jobs with squeue|command &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;]].&lt;br /&gt;
&lt;br /&gt;
== Partition availability ==&lt;br /&gt;
&lt;br /&gt;
The most important information that &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; provides is the &amp;#039;&amp;#039;&amp;#039;availability&amp;#039;&amp;#039;&amp;#039; (also called &amp;#039;&amp;#039;&amp;#039;state&amp;#039;&amp;#039;&amp;#039;) of partitions. This is shown in column &amp;quot;AVAIL&amp;quot;. Possible partition states are:&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is available&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed as soon as resources allow&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is in the process of becoming unavailable (i.e., of entering the &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; state: see below)&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is unavailable&lt;br /&gt;
:: There are no running jobs&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
When a partition goes from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; no harm is done to running jobs. In a normally functioning SLURM system, the passage from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; happens only when no jobs are running on the partition. If (e.g., due to a malfunction) the passage happens with jobs still running, they get killed.&lt;br /&gt;
&lt;br /&gt;
A partition in state &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; requires intervention by a [[Roles|Job Administrator]] to be restored to &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
== Default values ==&lt;br /&gt;
&lt;br /&gt;
The features of SLURM partitions, including the &amp;#039;&amp;#039;&amp;#039;default values&amp;#039;&amp;#039;&amp;#039; which are applied to jobs that do not make explicit requests, can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scontrol show partition&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PartitionName=jobs&lt;br /&gt;
   AllowGroups=ALL AllowAccounts=ALL AllowQos=nogpu,gpulight,gpu,gpuwide,gpuheavy-20,gpuheavy-40&lt;br /&gt;
   AllocNodes=ALL Default=YES QoS=N/A&lt;br /&gt;
=&amp;gt; DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO&lt;br /&gt;
   MaxNodes=UNLIMITED MaxTime=3-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED&lt;br /&gt;
   Nodes=gn01&lt;br /&gt;
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO&lt;br /&gt;
   OverTimeLimit=NONE PreemptMode=OFF&lt;br /&gt;
   State=UP TotalCPUs=48 TotalNodes=1 SelectTypeParameters=NONE&lt;br /&gt;
   JobDefaults=(null)&lt;br /&gt;
=&amp;gt; DefMemPerNode=4096 MaxMemPerNode=UNLIMITED&lt;br /&gt;
   TRES=cpu=48,mem=1011435M,node=1,billing=49,gres/gpu=13,gres/gpu:3g.20gb=5,gres/gpu:40gb=3,gres/gpu:4g.20gb=5&lt;br /&gt;
   TRESBillingWeights=cpu=1.0,gres/gpu:3g.20gb=6.0,gres/gpu:4g.20gb=6.0,gres/gpu:40gb=6.0,mem=0.05g&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In the example, we have highlighted with  &amp;#039;&amp;#039;&amp;#039;=&amp;gt;&amp;#039;&amp;#039;&amp;#039;  the most relevant default values for Mufasa users, i.e.:&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefaultTime&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default execution time assigned to a job run on the partition (e.g., 1 hour)&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefMemPerNode&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default amount of RAM assigned to a job run on the partition (e.g., 4GB)&lt;br /&gt;
&lt;br /&gt;
= System resources subjected to limitations =&lt;br /&gt;
&lt;br /&gt;
In systems based on SLURM like Mufasa, &amp;#039;&amp;#039;&amp;#039;TRES (Trackable RESources)&amp;#039;&amp;#039;&amp;#039; are (from [https://slurm.schedmd.com/tres.html SLURM&amp;#039;s documentation] &amp;quot;&amp;#039;&amp;#039;resources that can be tracked for usage or used to enforce limits against.&amp;#039;&amp;#039;&amp;quot;&lt;br /&gt;
&lt;br /&gt;
TRES include CPUs, RAM and &amp;#039;&amp;#039;&amp;#039;GRES&amp;#039;&amp;#039;&amp;#039;. The last term stands for &amp;#039;&amp;#039;Generic RESources&amp;#039;&amp;#039; that a job may need for its execution. In Mufasa, the only &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; resources are the GPUs.&lt;br /&gt;
&lt;br /&gt;
== &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax ==&lt;br /&gt;
&lt;br /&gt;
To ask SLURM to assign GRES resources (i.e., GPUs) to a job, a special syntax must be used. Precisely, the name of each GPU resource takes the form&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:Name:Type&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
Considering the [[System#CPUs and GPUs|GPU complement of Mufasa]], &amp;lt;code&amp;gt;Type&amp;lt;/code&amp;gt; takes the following values:&lt;br /&gt;
&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:40gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 40 Gbytes of RAM&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:4g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 4 compute units&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 3 compute units&lt;br /&gt;
&lt;br /&gt;
So, for instance,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
identifies a resource corresponding to a GPU with 20 GB of RAM and 3 compute units.&lt;br /&gt;
&lt;br /&gt;
When asking for a GRES resource (e.g., in an &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command or an &amp;lt;code&amp;gt;SBATCH&amp;lt;/code&amp;gt; directive of an [[User Jobs#Using execution scripts to run jobs|execution script]]), the syntax required by SLURM is&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:&amp;lt;Type&amp;gt;:&amp;lt;Quantity&amp;gt;&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
where &amp;lt;code&amp;gt;Quantity&amp;lt;/code&amp;gt; is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of type &amp;lt;code&amp;gt;4g.20gb&amp;lt;/code&amp;gt; the syntax is&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:4g.20gb:2&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SLURM&amp;#039;s &amp;#039;&amp;#039;generic resources&amp;#039;&amp;#039; are defined in &amp;lt;code&amp;gt;/etc/slurm/gres.conf&amp;lt;/code&amp;gt;. In order to make GPUs available to SLURM&amp;#039;s &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; management, Mufasa makes use of Nvidia&amp;#039;s [https://developer.nvidia.com/nvidia-management-library-nvml NVML library]. For additional information see [https://slurm.schedmd.com/gres.html SLURM&amp;#039;s documentation].&lt;/div&gt;</summary>
		<author><name>GiulioFontana</name></author>
	</entry>
	<entry>
		<id>https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2566</id>
		<title>SLURM</title>
		<link rel="alternate" type="text/html" href="https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2566"/>
		<updated>2026-11-03T14:27:48Z</updated>

		<summary type="html">&lt;p&gt;GiulioFontana: /* Elements determining job priority */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page presents the features of SLURM that are most relevant to Mufasa&amp;#039;s [[Roles|Job Users]]. Job Users can submit jobs for execution, cancel their own jobs, and see other users&amp;#039; jobs (but not intervene on them).&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa &amp;#039;&amp;#039;&amp;#039;must use SLURM&amp;#039;&amp;#039;&amp;#039; to run resource-heavy processes, i.e. computing jobs that require one or more of the following:&lt;br /&gt;
* GPUs&lt;br /&gt;
* multiple CPUs&lt;br /&gt;
* powerful CPUs&lt;br /&gt;
* a significant amount of RAM&lt;br /&gt;
&lt;br /&gt;
In fact, only processes run via SLURM have access to all the resources of Mufasa. Processes run outside SLURM are executed by the [[System#Login server|login server]] virtual machine, which has minimal resources and no GPUs. Using SLURM is therefore the only way to execute resource-heavy jobs on Mufasa (this is a key difference between Mufasa 1.0 and Mufasa 2.0).&lt;br /&gt;
&lt;br /&gt;
= SLURM in a nutshell =&lt;br /&gt;
&lt;br /&gt;
Computation jobs on Mufasa needs to be launched via [[System#The SLURM job scheduling system|SLURM]]. SLURM provides jobs with access to the [[#System resources subjected to limitations|physical resources]] of Mufasa, such as CPUs, GPUs and RAM. Thanks to SLURM, processing jobs share system resources, optimising their occupation and availability. &lt;br /&gt;
&lt;br /&gt;
When a user runs a job, the job does not get executed immediately and is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039;. SLURM executes jobs according to their order in the queue: the top job in the queue gets executed as soon as the necessary resources are available, while jobs lower in the queue wait longer. The position of a job in the queue is due to the &amp;#039;&amp;#039;&amp;#039;[[#Job priority|priority]]&amp;#039;&amp;#039;&amp;#039; assigned to it by SLURM, with higher-priority jobs closer to the top. As a general rule:&lt;br /&gt;
&lt;br /&gt;
;: &amp;#039;&amp;#039;&amp;#039;the greater the fraction of Mufasa&amp;#039;s overall resources that a job asks for, the lower the job&amp;#039;s priority will be&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
The priority mechanism is used to encourage users to use Mufasa&amp;#039;s resources (i.e.: GPUs, CPUs, RAM, execution time) in an effective and equitable manner. This page includes a [[#How_to_maximise_the_priority_of_your_jobs|chart explaining how to maximise the priority of your jobs]].&lt;br /&gt;
&lt;br /&gt;
The &amp;#039;&amp;#039;&amp;#039;time&amp;#039;&amp;#039;&amp;#039; available to a job for its execution is controlled by SLURM. When a user requests execution of a job, they must specify the duration of the time slot that the job needs. The job must complete its execution before the end of the requested time slot, because at the end of such slot it gets killed by SLURM.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0 access to system resources is managed via SLURM&amp;#039;s [[#SLURM Quality of Service (QOS)|Quality of Service (QOS)]] mechanism (Mufasa 1.0 used [[#SLURM_partitions|partitions]] instead). While [[User Jobs#Running jobs with SLURM|launching a processing job via SLURM]], the user must always specify the chosen QOS. QOSes differ in the set of resources that they provide access to because each of them is designed to fit a given type of job.&lt;br /&gt;
&lt;br /&gt;
Mufasa sets limits to the number of jobs by the same user. This page includes a [[#Limits on jobs by the same user|table summarising such limits]].&lt;br /&gt;
&lt;br /&gt;
= SLURM Quality of Service (QOS) =&lt;br /&gt;
&lt;br /&gt;
Through &amp;#039;&amp;#039;&amp;#039;Quality of Services&amp;#039;&amp;#039;&amp;#039; (&amp;#039;&amp;#039;&amp;#039;QOSes&amp;#039;&amp;#039;&amp;#039;), SLURM lets system configurators assign a name to a set of related constraints.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, QOSes are used to define different levels of access to the server&amp;#039;s resources. When [[User Jobs|executing a job with SLURM]], a user must always &amp;#039;&amp;#039;&amp;#039;specify the QOS&amp;#039;&amp;#039;&amp;#039; that their job will use: this choice, in turn, determines what resources the job is able to access and influences the [[#Job priority|priority]] of the job.&lt;br /&gt;
&lt;br /&gt;
Mufasa&amp;#039;s QOSes and their features can be inspected with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list qos format=name%-11,priority,MaxSubmit,maxwall,maxtres%-80,mintres%-18&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following (see below for the same data in table form):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
Name          Priority MaxSubmit     MaxWall MaxTRES                                                                          MinTRES            &lt;br /&gt;
----------- ---------- --------- ----------- -------------------------------------------------------------------------------- ------------------ &lt;br /&gt;
normal               0                                                                                                                           &lt;br /&gt;
nogpu                4         1  3-00:00:00 cpu=16,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=128G                    &lt;br /&gt;
gpuheavy-20          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=2,mem=128G             gres/gpu:4g.20gb=1 &lt;br /&gt;
gpuheavy-40          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=1,gres/gpu:4g.20gb=0,mem=128G             gres/gpu:40gb=1    &lt;br /&gt;
gpulight             8         1    12:00:00 cpu=2,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpu                  2         1  1-00:00:00 cpu=8,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpuwide              2         2  1-00:00:00 cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=1,mem=64G              gres/gpu:4g.20gb=1 &lt;br /&gt;
build               32         1    02:00:00 cpu=2,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=16G&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The columns of this output are the following:&lt;br /&gt;
&lt;br /&gt;
:; Name&lt;br /&gt;
:: name of the QOS&lt;br /&gt;
&lt;br /&gt;
:; Priority&lt;br /&gt;
:: priority tier associated to the QOS (higher value = higher priority): see [[#Job priority|Job priority]] for details&lt;br /&gt;
&lt;br /&gt;
:; MaxSubmit&lt;br /&gt;
:: maximum number of jobs from a single user that can be submitted to SLURM with this QOS; submitted jobs include both running and queued jobs&lt;br /&gt;
:: See [[#Limits on jobs by the same user|Limits on jobs by the same user]] for an overview of the limits on jobs set by Mufasa.&lt;br /&gt;
&lt;br /&gt;
:; MaxWall&lt;br /&gt;
:: maximum wall clock duration of the jobs using the QOS (after which they are killed by SLURM), in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
:: For some QOSes these are not set: it means that they are determined by the [[#SLURM partitions|partition]]. Partitions also define the [[#Default values|default duration]] of jobs.&lt;br /&gt;
&lt;br /&gt;
:; MaxTRES&lt;br /&gt;
:: amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) available to a job using the QOS, where&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;cpu=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of CPUs (i.e., processor cores) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of CPUs specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;gres/&amp;#039;&amp;#039;gpu:Type&amp;#039;&amp;#039;=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of GPUs of class &amp;lt;code&amp;gt;&amp;#039;&amp;#039;Type&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; (see [[User Jobs#gres syntax|&amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax]]) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; (for QOSes that allow access to GPUs) if not specified, the job cannot be launched&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;mem=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;G&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum amount of system RAM is &amp;#039;&amp;#039;K&amp;#039;&amp;#039; GBytes&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of RAM specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
&lt;br /&gt;
:; MinTRES&lt;br /&gt;
:: minimum amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) that a job using the QOS must request in order to actually get executed by SLURM.&lt;br /&gt;
:: (If your job does not actually need these resources, you&amp;#039;ve chosen the wrong QOS: you can use one with a higher priority.)&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; QOS is the default one, and exists only to ensure that users always specify a QOS when running a job. Since &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; has zero priority and no resources, a job run using this QOS would never be run.&lt;br /&gt;
&lt;br /&gt;
The information provided by the &amp;lt;code&amp;gt;sacctmgr list qos&amp;lt;/code&amp;gt; command above is summarised by the following table:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! QOS&lt;br /&gt;
! Priority&amp;lt;/br&amp;gt;tier&lt;br /&gt;
! Max&amp;lt;/br&amp;gt;Submit&lt;br /&gt;
! MaxWall&amp;lt;/br&amp;gt;[h]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;CPUs&lt;br /&gt;
! max RAM&amp;lt;/br&amp;gt;[GB]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;3g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;4g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;40GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | build&lt;br /&gt;
| 32&lt;br /&gt;
| 1&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 16&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpulight&lt;br /&gt;
| 8&lt;br /&gt;
| 1&lt;br /&gt;
| 12&lt;br /&gt;
| 2&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | nogpu&lt;br /&gt;
| 4&lt;br /&gt;
| 1&lt;br /&gt;
| 72&lt;br /&gt;
| 16&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpu&lt;br /&gt;
| 2&lt;br /&gt;
| 1&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuwide&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-20&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| 2 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-40&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
A key piece of information in the table above is the &amp;#039;&amp;#039;&amp;#039;priority tier&amp;#039;&amp;#039;&amp;#039; associated to each QOS. In Mufasa 2.0, priority tiers are used to encourage users to use the &amp;#039;&amp;#039;&amp;#039;least powerful QOS that is compatible with their needs&amp;#039;&amp;#039;&amp;#039;, where &amp;quot;powerful&amp;quot; means &amp;quot;rich with resources&amp;quot;. Less powerful QOSes increase the priority of the jobs that use them, so these jobs tend to be executed sooner. See [[#Job priority|Job priority]] for details about how priority affects the execution order of jobs in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
== The &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS ==&lt;br /&gt;
&lt;br /&gt;
This QOS is specifically designed to be used by Mufasa users to &amp;#039;&amp;#039;&amp;#039;quickly build [[System#Containers|container images]]&amp;#039;&amp;#039;&amp;#039;. Its associated priority tier is very high: this, combined with the fact that jobs using this QOS require few resources, means that such jobs usually get executed very soon.&lt;br /&gt;
&lt;br /&gt;
On the other hand, the limited resources and the lack of access to the GPUs make the &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS unsuitable for other tasks.&lt;br /&gt;
&lt;br /&gt;
See [[Singularity#Building Singularity images|Building Singularity images]] for directions about building Singularity container images.&lt;br /&gt;
&lt;br /&gt;
== Restricted QOSes ==&lt;br /&gt;
&lt;br /&gt;
In Mufasa, the most powerful QOSes are reserved to researchers (called &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users: these include academic personnel and Ph.D. students). M.Sc. students, called &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users, cannot use them while running jobs.&lt;br /&gt;
&lt;br /&gt;
See [[#research users and students users|below]] to understand how &amp;#039;&amp;#039;user categories&amp;#039;&amp;#039; work in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
= &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users =&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa belong to two &amp;#039;&amp;#039;&amp;#039;user categories&amp;#039;&amp;#039;&amp;#039;. Categories provide users with different access to  to Mufasa&amp;#039;s resources: the idea being to provide researchers with more access while still letting students benefit from the server&amp;#039;s capabilities.&lt;br /&gt;
&lt;br /&gt;
User categories in Mufasa are the following:&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. academic personnel and Ph.D. students&lt;br /&gt;
::: * can use all QOSes, including the the [[#Restricted QOSes|restricted ones]]&lt;br /&gt;
::: * their jobs have a higher &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a higher number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. M.Sc. students&lt;br /&gt;
::: * cannot use the [[#Restricted QOSes|restricted QOSes]]&lt;br /&gt;
::: * their jobs have a lower &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a lower number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
You can inspect the differences in priority and running jobs between &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association format=&amp;quot;account,priority,maxjobs&amp;quot; | grep -E &amp;#039;Account|research|students&amp;#039;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
   Account   Priority MaxJobs &lt;br /&gt;
  research          4       2 &lt;br /&gt;
  students          1       1&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
To know what limits apply to your own user, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association where user=$USER format=&amp;quot;user,priority,maxjobs,qos%-60&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
      User   Priority MaxJobs QOS                                                          &lt;br /&gt;
---------- ---------- ------- ------------------------------------------------------------ &lt;br /&gt;
    preali          4       2 build,gpu,gpuheavy-20,gpuheavy-40,gpulight,gpuwide,nogpu&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The list under &amp;quot;QOS&amp;quot; shows what QOSes your user is allowed to use when [[User Jobs|running jobs]]. &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users can use all of them, while &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users can only access a subset of them.&lt;br /&gt;
&lt;br /&gt;
= Limits on jobs by the same user =&lt;br /&gt;
&lt;br /&gt;
Mufasa enforces limits on the number of jobs from a single user. Such limits aim at preventing users from &amp;quot;hogging&amp;quot; system resources, and apply to:&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;submitted jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that the user asked SLURM to execute, each of which may currently be either running or queued&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;running jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that are currently in execution&lt;br /&gt;
&lt;br /&gt;
Limits on submitted jobs are set via [[#SLURM Quality of Service (QOS)|QOSes]], while limits on running jobs are set via [[#research users and students users|user category]].&lt;br /&gt;
&lt;br /&gt;
The following table summarises what limits exist:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! &lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;running&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;submitted&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | global limits&amp;lt;br/&amp;gt;(system-wide)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to limits set by individual QOSes (below)&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | limits for&amp;lt;br/&amp;gt;individual&amp;lt;/br&amp;gt;QOSes&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to global limits (above)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for the&amp;#039;&amp;#039; &amp;lt;code&amp;gt;gpuwide&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;QOS&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for each of the other QOSes&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
= Job priority =&lt;br /&gt;
&lt;br /&gt;
Once the execution of a job has been requested, the job is not run immediately: it is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039; by SLURM, together with all the other jobs awaiting execution. The job on top of the queue at any time is the first to be put into execution as soon as the resources it requires are available. &lt;br /&gt;
&lt;br /&gt;
The order of the items in the job queue depends on their &amp;#039;&amp;#039;&amp;#039;priority&amp;#039;&amp;#039;&amp;#039;. The job with the highest priority is put by SLURM on top of the queue, while all other jobs are put in the queue in order of descending priority.&lt;br /&gt;
&lt;br /&gt;
The goal of SLURM is to maximise resource availability: i.e., to ensure the shorter possible wait time before job execution. To achieve this goal, in Mufasa SLURM is configured to &amp;#039;&amp;#039;&amp;#039;encourage users not to ask for resources or execution time that their job doesn&amp;#039;t need&amp;#039;&amp;#039;&amp;#039;. This is done via the priority mechanism: the more resources and/or time a job requests, the lower its priority will be; and the later it will be executed.&lt;br /&gt;
&lt;br /&gt;
Priority management in Mufasa is designed to set up a &amp;#039;&amp;#039;&amp;#039;virtuous cycle&amp;#039;&amp;#039;&amp;#039; where users, by carefully choosing what to ask for, obtain two results:&lt;br /&gt;
* they ensure that their job is executed as soon as possible;&lt;br /&gt;
* they leave as much as possible of Mufasa&amp;#039;s resources free for other users&amp;#039;s jobs.&lt;br /&gt;
&lt;br /&gt;
== Elements determining job priority ==&lt;br /&gt;
In Mufasa, the priority of a job is computed by SLURM according to the following elements:&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#research users and students users|User category]]&amp;#039;&amp;#039;&amp;#039; (i.e., &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;)&lt;br /&gt;
::: Used to provide higher priority to jobs run by &amp;#039;&amp;#039;&amp;#039;research personnel&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#SLURM Quality of Service (QOS)|QOS]]&amp;#039;&amp;#039;&amp;#039; used by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;less resources&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Number of CPUs&amp;#039;&amp;#039;&amp;#039; requested by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;a lower number of CPUs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job Age&amp;#039;&amp;#039;&amp;#039;, i.e. the time that the job has been waiting in the queue&lt;br /&gt;
::: Used to provide higher priority to jobs which have been &amp;#039;&amp;#039;&amp;#039;queued for a longer time&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;FairShare&amp;#039;&amp;#039;&amp;#039;, i.e. a factor computed by SLURM to balance use of the system by different users&lt;br /&gt;
::: Used to provide higher priority to jobs by users who &amp;#039;&amp;#039;&amp;#039;used Mufasa&amp;#039;s resources less than others&amp;#039;&amp;#039;&amp;#039; (see [[#How FairShare works|below]])&lt;br /&gt;
&lt;br /&gt;
You can understand how each of these elements contribute to the priority of your jobs using command &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;. A tailored version of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; that only displays the parameters listed above is the following:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sprio -o &amp;quot;%6i %9u %.5Y %.10B %.5Q %.9T %.5A %.5F&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
Here is an example of its output:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
JOBID  USER      PRIOR      ASSOC   QOS      TRES   AGE FAIRS&lt;br /&gt;
5774   itombini   3492       2500   750  cpu=-800   906   136&lt;br /&gt;
5780   fbottazzi  2266        625   750  cpu=-600   420  1071&lt;br /&gt;
5783   mzoccola   1766        625   750  cpu=-800   217   974&lt;br /&gt;
5790   amoglia    2717       2500   375  cpu=-800   156   487&lt;br /&gt;
5791   qliu       2232       2500   375  cpu=-800   138    19&lt;br /&gt;
5792   fpastori   2197       2500   375  cpu=-800    84    39&lt;br /&gt;
5793   bagostini  2034        625  1500 cpu=-1200    18  1091&lt;br /&gt;
5794   lpassoni   2517       2500   750  cpu=-800     9    58&lt;br /&gt;
5795   lpassoni   2517       2500   750  cpu=-800     9    58&lt;br /&gt;
5796   adurat     3101       2500   750  cpu=-800     8   643&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
In it, column ASSOC (i.e., association) corresponds to user category and column TRES corresponds to the number of CPUs. Column PRIOR provides the overall priority of each job.&lt;br /&gt;
&lt;br /&gt;
=== How FairShare works ===&lt;br /&gt;
In Mufasa 2.0, the FairShare of each user is computed according to these rules:&lt;br /&gt;
# It considers the quantity used by (current and past) user jobs of:&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;CPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of CPUs used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;RAM&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the amount of RAM used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;GPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of GPUs used&lt;br /&gt;
#** note: 40 GB GPUs have double the impact on FairShare than 20 GB GPUs&lt;br /&gt;
# The impact on FairShare of any use of resources is &amp;#039;&amp;#039;&amp;#039;proportional to its duration&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
#* example: using 32 GB of RAM for 48h has the same impact of using 64 GB of RAM for 24h&lt;br /&gt;
# FairShare has a &amp;quot;fading memory&amp;quot;: i.e., resource use has less and less impact on FairShare the farther in the past it is&lt;br /&gt;
#* consequence: over time, the &amp;quot;history&amp;quot; of a user gets forgotten by FairShare&lt;br /&gt;
&lt;br /&gt;
== How to maximise the priority of your jobs ==&lt;br /&gt;
&lt;br /&gt;
Every time you run a SLURM job, follow these guidelines:&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Choose the less powerful QOS compatible with the needs of your job&lt;br /&gt;
:: QOSes with access to less resources have higher base priority&lt;br /&gt;
&lt;br /&gt;
; Only request CPUs and RAM that your job really needs&lt;br /&gt;
:: Usually, code exploits multiple CPUs only if specifically designed to do so: if unsure, only ask for 1 CPU&lt;br /&gt;
:: The fewer resources your job asks for, the less it will wait before they become available&lt;br /&gt;
:: Asking for fewer CPUs and/or less RAM improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Do not request more time than your jobs needs to complete&lt;br /&gt;
:: Make a worst-case estimate and only ask for that duration&lt;br /&gt;
:: Asking for less of Mufasa&amp;#039;s time improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Debug and test code using less powerful QOSes before running it with more powerful QOSes&lt;br /&gt;
:: Your test jobs will get a higher priority and your FairShare will improve&lt;br /&gt;
&lt;br /&gt;
; Cancel jobs that you don&amp;#039;t need them anymore&lt;br /&gt;
:: Always use [[User_Jobs#Cancelling_a_job_with_scancel|scancel]] to delete completed (or crashed) jobs: your Fairshare will improve&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Protip: it&amp;#039;s a good idea to [[User Jobs#Looking for unused GPUs|check for unused GPUs]] before choosing what to request. Requesting a GPU that is currently idle will help your job get executed sooner.&lt;br /&gt;
&lt;br /&gt;
= SLURM partitions =&lt;br /&gt;
&lt;br /&gt;
Partitions are another mechanism provided by SLURM to create different levels of access to system resources. Since in Mufasa 2.0 access to resources is controlled via [[#SLURM Quality of Service (QOS)|QOSes]], partitions are not very relevant. (This is one of the main differences between Mufasa 1.0 and Mufasa 2.0.)&lt;br /&gt;
&lt;br /&gt;
Note, however, that the default values for some features of SLURM jobs (e.g., duration) are [[#Default values|set by the partition]].&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, there is a single SLURM partition, called &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt;, and all jobs run on it. The state of &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt; can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sinfo -o &amp;quot;%10P %5a %9T %11L %10l&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PARTITION  AVAIL STATE     DEFAULTTIME TIMELIMIT &lt;br /&gt;
jobs*      up    idle      1:00:00     3-00:00:00&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where columns correspond to the following information:&lt;br /&gt;
&lt;br /&gt;
:; PARTITION&lt;br /&gt;
:: name of the partition; the asterisks indicates that it&amp;#039;s the default one&lt;br /&gt;
&lt;br /&gt;
:; AVAIL&lt;br /&gt;
:: state/availability of the partition: see [[#Partition availability|below]]&lt;br /&gt;
&lt;br /&gt;
:; STATE&lt;br /&gt;
:: state (using [https://slurm.schedmd.com/sinfo.html#SECTION_NODE-STATE-CODES these codes])&lt;br /&gt;
:: typical values are &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;mixed&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that some of the resources are busy executing jobs while other are idle, and &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;allocated&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that all of the resources are in use&lt;br /&gt;
&lt;br /&gt;
:; DEFAULTTIME&lt;br /&gt;
:: default runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:; TIMELIMIT&lt;br /&gt;
:: maximum runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
The asterisk at the end of the partition name indicates the default partition, i.e. the one on which jobs which do not ask for a specific partition are run.&lt;br /&gt;
&lt;br /&gt;
Command &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; does not tell you about the &amp;#039;&amp;#039;jobs&amp;#039;&amp;#039; submitted to a partition. This information is obtained, instead, with [[User Jobs#Inspecting jobs with squeue|command &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;]].&lt;br /&gt;
&lt;br /&gt;
== Partition availability ==&lt;br /&gt;
&lt;br /&gt;
The most important information that &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; provides is the &amp;#039;&amp;#039;&amp;#039;availability&amp;#039;&amp;#039;&amp;#039; (also called &amp;#039;&amp;#039;&amp;#039;state&amp;#039;&amp;#039;&amp;#039;) of partitions. This is shown in column &amp;quot;AVAIL&amp;quot;. Possible partition states are:&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is available&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed as soon as resources allow&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is in the process of becoming unavailable (i.e., of entering the &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; state: see below)&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is unavailable&lt;br /&gt;
:: There are no running jobs&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
When a partition goes from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; no harm is done to running jobs. In a normally functioning SLURM system, the passage from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; happens only when no jobs are running on the partition. If (e.g., due to a malfunction) the passage happens with jobs still running, they get killed.&lt;br /&gt;
&lt;br /&gt;
A partition in state &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; requires intervention by a [[Roles|Job Administrator]] to be restored to &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
== Default values ==&lt;br /&gt;
&lt;br /&gt;
The features of SLURM partitions, including the &amp;#039;&amp;#039;&amp;#039;default values&amp;#039;&amp;#039;&amp;#039; which are applied to jobs that do not make explicit requests, can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scontrol show partition&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PartitionName=jobs&lt;br /&gt;
   AllowGroups=ALL AllowAccounts=ALL AllowQos=nogpu,gpulight,gpu,gpuwide,gpuheavy-20,gpuheavy-40&lt;br /&gt;
   AllocNodes=ALL Default=YES QoS=N/A&lt;br /&gt;
=&amp;gt; DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO&lt;br /&gt;
   MaxNodes=UNLIMITED MaxTime=3-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED&lt;br /&gt;
   Nodes=gn01&lt;br /&gt;
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO&lt;br /&gt;
   OverTimeLimit=NONE PreemptMode=OFF&lt;br /&gt;
   State=UP TotalCPUs=48 TotalNodes=1 SelectTypeParameters=NONE&lt;br /&gt;
   JobDefaults=(null)&lt;br /&gt;
=&amp;gt; DefMemPerNode=4096 MaxMemPerNode=UNLIMITED&lt;br /&gt;
   TRES=cpu=48,mem=1011435M,node=1,billing=49,gres/gpu=13,gres/gpu:3g.20gb=5,gres/gpu:40gb=3,gres/gpu:4g.20gb=5&lt;br /&gt;
   TRESBillingWeights=cpu=1.0,gres/gpu:3g.20gb=6.0,gres/gpu:4g.20gb=6.0,gres/gpu:40gb=6.0,mem=0.05g&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In the example, we have highlighted with  &amp;#039;&amp;#039;&amp;#039;=&amp;gt;&amp;#039;&amp;#039;&amp;#039;  the most relevant default values for Mufasa users, i.e.:&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefaultTime&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default execution time assigned to a job run on the partition (e.g., 1 hour)&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefMemPerNode&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default amount of RAM assigned to a job run on the partition (e.g., 4GB)&lt;br /&gt;
&lt;br /&gt;
= System resources subjected to limitations =&lt;br /&gt;
&lt;br /&gt;
In systems based on SLURM like Mufasa, &amp;#039;&amp;#039;&amp;#039;TRES (Trackable RESources)&amp;#039;&amp;#039;&amp;#039; are (from [https://slurm.schedmd.com/tres.html SLURM&amp;#039;s documentation] &amp;quot;&amp;#039;&amp;#039;resources that can be tracked for usage or used to enforce limits against.&amp;#039;&amp;#039;&amp;quot;&lt;br /&gt;
&lt;br /&gt;
TRES include CPUs, RAM and &amp;#039;&amp;#039;&amp;#039;GRES&amp;#039;&amp;#039;&amp;#039;. The last term stands for &amp;#039;&amp;#039;Generic RESources&amp;#039;&amp;#039; that a job may need for its execution. In Mufasa, the only &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; resources are the GPUs.&lt;br /&gt;
&lt;br /&gt;
== &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax ==&lt;br /&gt;
&lt;br /&gt;
To ask SLURM to assign GRES resources (i.e., GPUs) to a job, a special syntax must be used. Precisely, the name of each GPU resource takes the form&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:Name:Type&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
Considering the [[System#CPUs and GPUs|GPU complement of Mufasa]], &amp;lt;code&amp;gt;Type&amp;lt;/code&amp;gt; takes the following values:&lt;br /&gt;
&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:40gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 40 Gbytes of RAM&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:4g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 4 compute units&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 3 compute units&lt;br /&gt;
&lt;br /&gt;
So, for instance,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
identifies a resource corresponding to a GPU with 20 GB of RAM and 3 compute units.&lt;br /&gt;
&lt;br /&gt;
When asking for a GRES resource (e.g., in an &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command or an &amp;lt;code&amp;gt;SBATCH&amp;lt;/code&amp;gt; directive of an [[User Jobs#Using execution scripts to run jobs|execution script]]), the syntax required by SLURM is&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:&amp;lt;Type&amp;gt;:&amp;lt;Quantity&amp;gt;&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
where &amp;lt;code&amp;gt;Quantity&amp;lt;/code&amp;gt; is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of type &amp;lt;code&amp;gt;4g.20gb&amp;lt;/code&amp;gt; the syntax is&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:4g.20gb:2&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SLURM&amp;#039;s &amp;#039;&amp;#039;generic resources&amp;#039;&amp;#039; are defined in &amp;lt;code&amp;gt;/etc/slurm/gres.conf&amp;lt;/code&amp;gt;. In order to make GPUs available to SLURM&amp;#039;s &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; management, Mufasa makes use of Nvidia&amp;#039;s [https://developer.nvidia.com/nvidia-management-library-nvml NVML library]. For additional information see [https://slurm.schedmd.com/gres.html SLURM&amp;#039;s documentation].&lt;/div&gt;</summary>
		<author><name>GiulioFontana</name></author>
	</entry>
	<entry>
		<id>https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2565</id>
		<title>SLURM</title>
		<link rel="alternate" type="text/html" href="https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2565"/>
		<updated>2026-11-03T14:21:51Z</updated>

		<summary type="html">&lt;p&gt;GiulioFontana: /* Elements determining job priority */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page presents the features of SLURM that are most relevant to Mufasa&amp;#039;s [[Roles|Job Users]]. Job Users can submit jobs for execution, cancel their own jobs, and see other users&amp;#039; jobs (but not intervene on them).&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa &amp;#039;&amp;#039;&amp;#039;must use SLURM&amp;#039;&amp;#039;&amp;#039; to run resource-heavy processes, i.e. computing jobs that require one or more of the following:&lt;br /&gt;
* GPUs&lt;br /&gt;
* multiple CPUs&lt;br /&gt;
* powerful CPUs&lt;br /&gt;
* a significant amount of RAM&lt;br /&gt;
&lt;br /&gt;
In fact, only processes run via SLURM have access to all the resources of Mufasa. Processes run outside SLURM are executed by the [[System#Login server|login server]] virtual machine, which has minimal resources and no GPUs. Using SLURM is therefore the only way to execute resource-heavy jobs on Mufasa (this is a key difference between Mufasa 1.0 and Mufasa 2.0).&lt;br /&gt;
&lt;br /&gt;
= SLURM in a nutshell =&lt;br /&gt;
&lt;br /&gt;
Computation jobs on Mufasa needs to be launched via [[System#The SLURM job scheduling system|SLURM]]. SLURM provides jobs with access to the [[#System resources subjected to limitations|physical resources]] of Mufasa, such as CPUs, GPUs and RAM. Thanks to SLURM, processing jobs share system resources, optimising their occupation and availability. &lt;br /&gt;
&lt;br /&gt;
When a user runs a job, the job does not get executed immediately and is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039;. SLURM executes jobs according to their order in the queue: the top job in the queue gets executed as soon as the necessary resources are available, while jobs lower in the queue wait longer. The position of a job in the queue is due to the &amp;#039;&amp;#039;&amp;#039;[[#Job priority|priority]]&amp;#039;&amp;#039;&amp;#039; assigned to it by SLURM, with higher-priority jobs closer to the top. As a general rule:&lt;br /&gt;
&lt;br /&gt;
;: &amp;#039;&amp;#039;&amp;#039;the greater the fraction of Mufasa&amp;#039;s overall resources that a job asks for, the lower the job&amp;#039;s priority will be&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
The priority mechanism is used to encourage users to use Mufasa&amp;#039;s resources (i.e.: GPUs, CPUs, RAM, execution time) in an effective and equitable manner. This page includes a [[#How_to_maximise_the_priority_of_your_jobs|chart explaining how to maximise the priority of your jobs]].&lt;br /&gt;
&lt;br /&gt;
The &amp;#039;&amp;#039;&amp;#039;time&amp;#039;&amp;#039;&amp;#039; available to a job for its execution is controlled by SLURM. When a user requests execution of a job, they must specify the duration of the time slot that the job needs. The job must complete its execution before the end of the requested time slot, because at the end of such slot it gets killed by SLURM.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0 access to system resources is managed via SLURM&amp;#039;s [[#SLURM Quality of Service (QOS)|Quality of Service (QOS)]] mechanism (Mufasa 1.0 used [[#SLURM_partitions|partitions]] instead). While [[User Jobs#Running jobs with SLURM|launching a processing job via SLURM]], the user must always specify the chosen QOS. QOSes differ in the set of resources that they provide access to because each of them is designed to fit a given type of job.&lt;br /&gt;
&lt;br /&gt;
Mufasa sets limits to the number of jobs by the same user. This page includes a [[#Limits on jobs by the same user|table summarising such limits]].&lt;br /&gt;
&lt;br /&gt;
= SLURM Quality of Service (QOS) =&lt;br /&gt;
&lt;br /&gt;
Through &amp;#039;&amp;#039;&amp;#039;Quality of Services&amp;#039;&amp;#039;&amp;#039; (&amp;#039;&amp;#039;&amp;#039;QOSes&amp;#039;&amp;#039;&amp;#039;), SLURM lets system configurators assign a name to a set of related constraints.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, QOSes are used to define different levels of access to the server&amp;#039;s resources. When [[User Jobs|executing a job with SLURM]], a user must always &amp;#039;&amp;#039;&amp;#039;specify the QOS&amp;#039;&amp;#039;&amp;#039; that their job will use: this choice, in turn, determines what resources the job is able to access and influences the [[#Job priority|priority]] of the job.&lt;br /&gt;
&lt;br /&gt;
Mufasa&amp;#039;s QOSes and their features can be inspected with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list qos format=name%-11,priority,MaxSubmit,maxwall,maxtres%-80,mintres%-18&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following (see below for the same data in table form):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
Name          Priority MaxSubmit     MaxWall MaxTRES                                                                          MinTRES            &lt;br /&gt;
----------- ---------- --------- ----------- -------------------------------------------------------------------------------- ------------------ &lt;br /&gt;
normal               0                                                                                                                           &lt;br /&gt;
nogpu                4         1  3-00:00:00 cpu=16,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=128G                    &lt;br /&gt;
gpuheavy-20          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=2,mem=128G             gres/gpu:4g.20gb=1 &lt;br /&gt;
gpuheavy-40          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=1,gres/gpu:4g.20gb=0,mem=128G             gres/gpu:40gb=1    &lt;br /&gt;
gpulight             8         1    12:00:00 cpu=2,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpu                  2         1  1-00:00:00 cpu=8,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpuwide              2         2  1-00:00:00 cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=1,mem=64G              gres/gpu:4g.20gb=1 &lt;br /&gt;
build               32         1    02:00:00 cpu=2,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=16G&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The columns of this output are the following:&lt;br /&gt;
&lt;br /&gt;
:; Name&lt;br /&gt;
:: name of the QOS&lt;br /&gt;
&lt;br /&gt;
:; Priority&lt;br /&gt;
:: priority tier associated to the QOS (higher value = higher priority): see [[#Job priority|Job priority]] for details&lt;br /&gt;
&lt;br /&gt;
:; MaxSubmit&lt;br /&gt;
:: maximum number of jobs from a single user that can be submitted to SLURM with this QOS; submitted jobs include both running and queued jobs&lt;br /&gt;
:: See [[#Limits on jobs by the same user|Limits on jobs by the same user]] for an overview of the limits on jobs set by Mufasa.&lt;br /&gt;
&lt;br /&gt;
:; MaxWall&lt;br /&gt;
:: maximum wall clock duration of the jobs using the QOS (after which they are killed by SLURM), in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
:: For some QOSes these are not set: it means that they are determined by the [[#SLURM partitions|partition]]. Partitions also define the [[#Default values|default duration]] of jobs.&lt;br /&gt;
&lt;br /&gt;
:; MaxTRES&lt;br /&gt;
:: amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) available to a job using the QOS, where&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;cpu=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of CPUs (i.e., processor cores) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of CPUs specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;gres/&amp;#039;&amp;#039;gpu:Type&amp;#039;&amp;#039;=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of GPUs of class &amp;lt;code&amp;gt;&amp;#039;&amp;#039;Type&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; (see [[User Jobs#gres syntax|&amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax]]) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; (for QOSes that allow access to GPUs) if not specified, the job cannot be launched&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;mem=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;G&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum amount of system RAM is &amp;#039;&amp;#039;K&amp;#039;&amp;#039; GBytes&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of RAM specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
&lt;br /&gt;
:; MinTRES&lt;br /&gt;
:: minimum amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) that a job using the QOS must request in order to actually get executed by SLURM.&lt;br /&gt;
:: (If your job does not actually need these resources, you&amp;#039;ve chosen the wrong QOS: you can use one with a higher priority.)&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; QOS is the default one, and exists only to ensure that users always specify a QOS when running a job. Since &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; has zero priority and no resources, a job run using this QOS would never be run.&lt;br /&gt;
&lt;br /&gt;
The information provided by the &amp;lt;code&amp;gt;sacctmgr list qos&amp;lt;/code&amp;gt; command above is summarised by the following table:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! QOS&lt;br /&gt;
! Priority&amp;lt;/br&amp;gt;tier&lt;br /&gt;
! Max&amp;lt;/br&amp;gt;Submit&lt;br /&gt;
! MaxWall&amp;lt;/br&amp;gt;[h]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;CPUs&lt;br /&gt;
! max RAM&amp;lt;/br&amp;gt;[GB]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;3g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;4g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;40GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | build&lt;br /&gt;
| 32&lt;br /&gt;
| 1&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 16&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpulight&lt;br /&gt;
| 8&lt;br /&gt;
| 1&lt;br /&gt;
| 12&lt;br /&gt;
| 2&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | nogpu&lt;br /&gt;
| 4&lt;br /&gt;
| 1&lt;br /&gt;
| 72&lt;br /&gt;
| 16&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpu&lt;br /&gt;
| 2&lt;br /&gt;
| 1&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuwide&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-20&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| 2 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-40&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
A key piece of information in the table above is the &amp;#039;&amp;#039;&amp;#039;priority tier&amp;#039;&amp;#039;&amp;#039; associated to each QOS. In Mufasa 2.0, priority tiers are used to encourage users to use the &amp;#039;&amp;#039;&amp;#039;least powerful QOS that is compatible with their needs&amp;#039;&amp;#039;&amp;#039;, where &amp;quot;powerful&amp;quot; means &amp;quot;rich with resources&amp;quot;. Less powerful QOSes increase the priority of the jobs that use them, so these jobs tend to be executed sooner. See [[#Job priority|Job priority]] for details about how priority affects the execution order of jobs in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
== The &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS ==&lt;br /&gt;
&lt;br /&gt;
This QOS is specifically designed to be used by Mufasa users to &amp;#039;&amp;#039;&amp;#039;quickly build [[System#Containers|container images]]&amp;#039;&amp;#039;&amp;#039;. Its associated priority tier is very high: this, combined with the fact that jobs using this QOS require few resources, means that such jobs usually get executed very soon.&lt;br /&gt;
&lt;br /&gt;
On the other hand, the limited resources and the lack of access to the GPUs make the &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS unsuitable for other tasks.&lt;br /&gt;
&lt;br /&gt;
See [[Singularity#Building Singularity images|Building Singularity images]] for directions about building Singularity container images.&lt;br /&gt;
&lt;br /&gt;
== Restricted QOSes ==&lt;br /&gt;
&lt;br /&gt;
In Mufasa, the most powerful QOSes are reserved to researchers (called &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users: these include academic personnel and Ph.D. students). M.Sc. students, called &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users, cannot use them while running jobs.&lt;br /&gt;
&lt;br /&gt;
See [[#research users and students users|below]] to understand how &amp;#039;&amp;#039;user categories&amp;#039;&amp;#039; work in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
= &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users =&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa belong to two &amp;#039;&amp;#039;&amp;#039;user categories&amp;#039;&amp;#039;&amp;#039;. Categories provide users with different access to  to Mufasa&amp;#039;s resources: the idea being to provide researchers with more access while still letting students benefit from the server&amp;#039;s capabilities.&lt;br /&gt;
&lt;br /&gt;
User categories in Mufasa are the following:&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. academic personnel and Ph.D. students&lt;br /&gt;
::: * can use all QOSes, including the the [[#Restricted QOSes|restricted ones]]&lt;br /&gt;
::: * their jobs have a higher &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a higher number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. M.Sc. students&lt;br /&gt;
::: * cannot use the [[#Restricted QOSes|restricted QOSes]]&lt;br /&gt;
::: * their jobs have a lower &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a lower number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
You can inspect the differences in priority and running jobs between &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association format=&amp;quot;account,priority,maxjobs&amp;quot; | grep -E &amp;#039;Account|research|students&amp;#039;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
   Account   Priority MaxJobs &lt;br /&gt;
  research          4       2 &lt;br /&gt;
  students          1       1&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
To know what limits apply to your own user, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association where user=$USER format=&amp;quot;user,priority,maxjobs,qos%-60&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
      User   Priority MaxJobs QOS                                                          &lt;br /&gt;
---------- ---------- ------- ------------------------------------------------------------ &lt;br /&gt;
    preali          4       2 build,gpu,gpuheavy-20,gpuheavy-40,gpulight,gpuwide,nogpu&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The list under &amp;quot;QOS&amp;quot; shows what QOSes your user is allowed to use when [[User Jobs|running jobs]]. &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users can use all of them, while &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users can only access a subset of them.&lt;br /&gt;
&lt;br /&gt;
= Limits on jobs by the same user =&lt;br /&gt;
&lt;br /&gt;
Mufasa enforces limits on the number of jobs from a single user. Such limits aim at preventing users from &amp;quot;hogging&amp;quot; system resources, and apply to:&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;submitted jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that the user asked SLURM to execute, each of which may currently be either running or queued&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;running jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that are currently in execution&lt;br /&gt;
&lt;br /&gt;
Limits on submitted jobs are set via [[#SLURM Quality of Service (QOS)|QOSes]], while limits on running jobs are set via [[#research users and students users|user category]].&lt;br /&gt;
&lt;br /&gt;
The following table summarises what limits exist:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! &lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;running&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;submitted&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | global limits&amp;lt;br/&amp;gt;(system-wide)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to limits set by individual QOSes (below)&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | limits for&amp;lt;br/&amp;gt;individual&amp;lt;/br&amp;gt;QOSes&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to global limits (above)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for the&amp;#039;&amp;#039; &amp;lt;code&amp;gt;gpuwide&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;QOS&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for each of the other QOSes&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
= Job priority =&lt;br /&gt;
&lt;br /&gt;
Once the execution of a job has been requested, the job is not run immediately: it is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039; by SLURM, together with all the other jobs awaiting execution. The job on top of the queue at any time is the first to be put into execution as soon as the resources it requires are available. &lt;br /&gt;
&lt;br /&gt;
The order of the items in the job queue depends on their &amp;#039;&amp;#039;&amp;#039;priority&amp;#039;&amp;#039;&amp;#039;. The job with the highest priority is put by SLURM on top of the queue, while all other jobs are put in the queue in order of descending priority.&lt;br /&gt;
&lt;br /&gt;
The goal of SLURM is to maximise resource availability: i.e., to ensure the shorter possible wait time before job execution. To achieve this goal, in Mufasa SLURM is configured to &amp;#039;&amp;#039;&amp;#039;encourage users not to ask for resources or execution time that their job doesn&amp;#039;t need&amp;#039;&amp;#039;&amp;#039;. This is done via the priority mechanism: the more resources and/or time a job requests, the lower its priority will be; and the later it will be executed.&lt;br /&gt;
&lt;br /&gt;
Priority management in Mufasa is designed to set up a &amp;#039;&amp;#039;&amp;#039;virtuous cycle&amp;#039;&amp;#039;&amp;#039; where users, by carefully choosing what to ask for, obtain two results:&lt;br /&gt;
* they ensure that their job is executed as soon as possible;&lt;br /&gt;
* they leave as much as possible of Mufasa&amp;#039;s resources free for other users&amp;#039;s jobs.&lt;br /&gt;
&lt;br /&gt;
== Elements determining job priority ==&lt;br /&gt;
In Mufasa, the priority of a job is computed by SLURM according to the following elements:&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#research users and students users|User category]]&amp;#039;&amp;#039;&amp;#039; (i.e., &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;)&lt;br /&gt;
::: Used to provide higher priority to jobs run by &amp;#039;&amp;#039;&amp;#039;research personnel&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#SLURM Quality of Service (QOS)|QOS]]&amp;#039;&amp;#039;&amp;#039; used by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;less resources&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Number of CPUs&amp;#039;&amp;#039;&amp;#039; requested by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;a lower number of CPUs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job Age&amp;#039;&amp;#039;&amp;#039;, i.e. the time that the job has been waiting in the queue&lt;br /&gt;
::: Used to provide higher priority to jobs which have been &amp;#039;&amp;#039;&amp;#039;queued for a longer time&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;FairShare&amp;#039;&amp;#039;&amp;#039;, i.e. a factor computed by SLURM to balance use of the system by different users&lt;br /&gt;
::: Used to provide higher priority to jobs by users who &amp;#039;&amp;#039;&amp;#039;used Mufasa&amp;#039;s resources less than others&amp;#039;&amp;#039;&amp;#039; (see [[#How FairShare works|below]])&lt;br /&gt;
&lt;br /&gt;
You can understand how each of these elements contribute to the priority of your jobs using command &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;. A tailored version of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; that only displays the parameters listed above is the following:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sprio -o &amp;quot;%6i %9u %.5Y %.5A %.5B %.5F %.5Q%.10T&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
Here is an example of its output:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
JOBID  USER      PRIOR   AGE ASSOC FAIRS   QOS      TRES&lt;br /&gt;
5774   itombini   3485   899  2500   136   750  cpu=-800&lt;br /&gt;
5780   fbottazzi  2259   413   625  1071   750  cpu=-600&lt;br /&gt;
5783   mzoccola   1759   210   625   974   750  cpu=-800&lt;br /&gt;
5790   amoglia    2710   149  2500   487   375  cpu=-800&lt;br /&gt;
5791   qliu       2225   131  2500    19   375  cpu=-800&lt;br /&gt;
5792   fpastori   2190    77  2500    39   375  cpu=-800&lt;br /&gt;
5793   bagostini  2027    11   625  1091  1500 cpu=-1200&lt;br /&gt;
5794   lpassoni   2510     2  2500    58   750  cpu=-800&lt;br /&gt;
5795   lpassoni   2510     2  2500    58   750  cpu=-800&lt;br /&gt;
5796   adurat     3094     2  2500   643   750  cpu=-800&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
In it, column ASSOC (i.e., association) corresponds to user category and column TRES corresponds to the number of CPUs. Column PRIOR provides the overall priority of each job.&lt;br /&gt;
&lt;br /&gt;
=== How FairShare works ===&lt;br /&gt;
In Mufasa 2.0, the FairShare of each user is computed according to these rules:&lt;br /&gt;
# It considers the quantity used by (current and past) user jobs of:&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;CPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of CPUs used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;RAM&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the amount of RAM used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;GPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of GPUs used&lt;br /&gt;
#** note: 40 GB GPUs have double the impact on FairShare than 20 GB GPUs&lt;br /&gt;
# The impact on FairShare of any use of resources is &amp;#039;&amp;#039;&amp;#039;proportional to its duration&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
#* example: using 32 GB of RAM for 48h has the same impact of using 64 GB of RAM for 24h&lt;br /&gt;
# FairShare has a &amp;quot;fading memory&amp;quot;: i.e., resource use has less and less impact on FairShare the farther in the past it is&lt;br /&gt;
#* consequence: over time, the &amp;quot;history&amp;quot; of a user gets forgotten by FairShare&lt;br /&gt;
&lt;br /&gt;
== How to maximise the priority of your jobs ==&lt;br /&gt;
&lt;br /&gt;
Every time you run a SLURM job, follow these guidelines:&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Choose the less powerful QOS compatible with the needs of your job&lt;br /&gt;
:: QOSes with access to less resources have higher base priority&lt;br /&gt;
&lt;br /&gt;
; Only request CPUs and RAM that your job really needs&lt;br /&gt;
:: Usually, code exploits multiple CPUs only if specifically designed to do so: if unsure, only ask for 1 CPU&lt;br /&gt;
:: The fewer resources your job asks for, the less it will wait before they become available&lt;br /&gt;
:: Asking for fewer CPUs and/or less RAM improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Do not request more time than your jobs needs to complete&lt;br /&gt;
:: Make a worst-case estimate and only ask for that duration&lt;br /&gt;
:: Asking for less of Mufasa&amp;#039;s time improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Debug and test code using less powerful QOSes before running it with more powerful QOSes&lt;br /&gt;
:: Your test jobs will get a higher priority and your FairShare will improve&lt;br /&gt;
&lt;br /&gt;
; Cancel jobs that you don&amp;#039;t need them anymore&lt;br /&gt;
:: Always use [[User_Jobs#Cancelling_a_job_with_scancel|scancel]] to delete completed (or crashed) jobs: your Fairshare will improve&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Protip: it&amp;#039;s a good idea to [[User Jobs#Looking for unused GPUs|check for unused GPUs]] before choosing what to request. Requesting a GPU that is currently idle will help your job get executed sooner.&lt;br /&gt;
&lt;br /&gt;
= SLURM partitions =&lt;br /&gt;
&lt;br /&gt;
Partitions are another mechanism provided by SLURM to create different levels of access to system resources. Since in Mufasa 2.0 access to resources is controlled via [[#SLURM Quality of Service (QOS)|QOSes]], partitions are not very relevant. (This is one of the main differences between Mufasa 1.0 and Mufasa 2.0.)&lt;br /&gt;
&lt;br /&gt;
Note, however, that the default values for some features of SLURM jobs (e.g., duration) are [[#Default values|set by the partition]].&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, there is a single SLURM partition, called &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt;, and all jobs run on it. The state of &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt; can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sinfo -o &amp;quot;%10P %5a %9T %11L %10l&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PARTITION  AVAIL STATE     DEFAULTTIME TIMELIMIT &lt;br /&gt;
jobs*      up    idle      1:00:00     3-00:00:00&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where columns correspond to the following information:&lt;br /&gt;
&lt;br /&gt;
:; PARTITION&lt;br /&gt;
:: name of the partition; the asterisks indicates that it&amp;#039;s the default one&lt;br /&gt;
&lt;br /&gt;
:; AVAIL&lt;br /&gt;
:: state/availability of the partition: see [[#Partition availability|below]]&lt;br /&gt;
&lt;br /&gt;
:; STATE&lt;br /&gt;
:: state (using [https://slurm.schedmd.com/sinfo.html#SECTION_NODE-STATE-CODES these codes])&lt;br /&gt;
:: typical values are &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;mixed&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that some of the resources are busy executing jobs while other are idle, and &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;allocated&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that all of the resources are in use&lt;br /&gt;
&lt;br /&gt;
:; DEFAULTTIME&lt;br /&gt;
:: default runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:; TIMELIMIT&lt;br /&gt;
:: maximum runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
The asterisk at the end of the partition name indicates the default partition, i.e. the one on which jobs which do not ask for a specific partition are run.&lt;br /&gt;
&lt;br /&gt;
Command &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; does not tell you about the &amp;#039;&amp;#039;jobs&amp;#039;&amp;#039; submitted to a partition. This information is obtained, instead, with [[User Jobs#Inspecting jobs with squeue|command &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;]].&lt;br /&gt;
&lt;br /&gt;
== Partition availability ==&lt;br /&gt;
&lt;br /&gt;
The most important information that &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; provides is the &amp;#039;&amp;#039;&amp;#039;availability&amp;#039;&amp;#039;&amp;#039; (also called &amp;#039;&amp;#039;&amp;#039;state&amp;#039;&amp;#039;&amp;#039;) of partitions. This is shown in column &amp;quot;AVAIL&amp;quot;. Possible partition states are:&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is available&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed as soon as resources allow&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is in the process of becoming unavailable (i.e., of entering the &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; state: see below)&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is unavailable&lt;br /&gt;
:: There are no running jobs&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
When a partition goes from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; no harm is done to running jobs. In a normally functioning SLURM system, the passage from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; happens only when no jobs are running on the partition. If (e.g., due to a malfunction) the passage happens with jobs still running, they get killed.&lt;br /&gt;
&lt;br /&gt;
A partition in state &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; requires intervention by a [[Roles|Job Administrator]] to be restored to &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
== Default values ==&lt;br /&gt;
&lt;br /&gt;
The features of SLURM partitions, including the &amp;#039;&amp;#039;&amp;#039;default values&amp;#039;&amp;#039;&amp;#039; which are applied to jobs that do not make explicit requests, can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scontrol show partition&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PartitionName=jobs&lt;br /&gt;
   AllowGroups=ALL AllowAccounts=ALL AllowQos=nogpu,gpulight,gpu,gpuwide,gpuheavy-20,gpuheavy-40&lt;br /&gt;
   AllocNodes=ALL Default=YES QoS=N/A&lt;br /&gt;
=&amp;gt; DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO&lt;br /&gt;
   MaxNodes=UNLIMITED MaxTime=3-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED&lt;br /&gt;
   Nodes=gn01&lt;br /&gt;
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO&lt;br /&gt;
   OverTimeLimit=NONE PreemptMode=OFF&lt;br /&gt;
   State=UP TotalCPUs=48 TotalNodes=1 SelectTypeParameters=NONE&lt;br /&gt;
   JobDefaults=(null)&lt;br /&gt;
=&amp;gt; DefMemPerNode=4096 MaxMemPerNode=UNLIMITED&lt;br /&gt;
   TRES=cpu=48,mem=1011435M,node=1,billing=49,gres/gpu=13,gres/gpu:3g.20gb=5,gres/gpu:40gb=3,gres/gpu:4g.20gb=5&lt;br /&gt;
   TRESBillingWeights=cpu=1.0,gres/gpu:3g.20gb=6.0,gres/gpu:4g.20gb=6.0,gres/gpu:40gb=6.0,mem=0.05g&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In the example, we have highlighted with  &amp;#039;&amp;#039;&amp;#039;=&amp;gt;&amp;#039;&amp;#039;&amp;#039;  the most relevant default values for Mufasa users, i.e.:&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefaultTime&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default execution time assigned to a job run on the partition (e.g., 1 hour)&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefMemPerNode&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default amount of RAM assigned to a job run on the partition (e.g., 4GB)&lt;br /&gt;
&lt;br /&gt;
= System resources subjected to limitations =&lt;br /&gt;
&lt;br /&gt;
In systems based on SLURM like Mufasa, &amp;#039;&amp;#039;&amp;#039;TRES (Trackable RESources)&amp;#039;&amp;#039;&amp;#039; are (from [https://slurm.schedmd.com/tres.html SLURM&amp;#039;s documentation] &amp;quot;&amp;#039;&amp;#039;resources that can be tracked for usage or used to enforce limits against.&amp;#039;&amp;#039;&amp;quot;&lt;br /&gt;
&lt;br /&gt;
TRES include CPUs, RAM and &amp;#039;&amp;#039;&amp;#039;GRES&amp;#039;&amp;#039;&amp;#039;. The last term stands for &amp;#039;&amp;#039;Generic RESources&amp;#039;&amp;#039; that a job may need for its execution. In Mufasa, the only &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; resources are the GPUs.&lt;br /&gt;
&lt;br /&gt;
== &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax ==&lt;br /&gt;
&lt;br /&gt;
To ask SLURM to assign GRES resources (i.e., GPUs) to a job, a special syntax must be used. Precisely, the name of each GPU resource takes the form&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:Name:Type&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
Considering the [[System#CPUs and GPUs|GPU complement of Mufasa]], &amp;lt;code&amp;gt;Type&amp;lt;/code&amp;gt; takes the following values:&lt;br /&gt;
&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:40gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 40 Gbytes of RAM&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:4g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 4 compute units&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 3 compute units&lt;br /&gt;
&lt;br /&gt;
So, for instance,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
identifies a resource corresponding to a GPU with 20 GB of RAM and 3 compute units.&lt;br /&gt;
&lt;br /&gt;
When asking for a GRES resource (e.g., in an &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command or an &amp;lt;code&amp;gt;SBATCH&amp;lt;/code&amp;gt; directive of an [[User Jobs#Using execution scripts to run jobs|execution script]]), the syntax required by SLURM is&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:&amp;lt;Type&amp;gt;:&amp;lt;Quantity&amp;gt;&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
where &amp;lt;code&amp;gt;Quantity&amp;lt;/code&amp;gt; is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of type &amp;lt;code&amp;gt;4g.20gb&amp;lt;/code&amp;gt; the syntax is&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:4g.20gb:2&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SLURM&amp;#039;s &amp;#039;&amp;#039;generic resources&amp;#039;&amp;#039; are defined in &amp;lt;code&amp;gt;/etc/slurm/gres.conf&amp;lt;/code&amp;gt;. In order to make GPUs available to SLURM&amp;#039;s &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; management, Mufasa makes use of Nvidia&amp;#039;s [https://developer.nvidia.com/nvidia-management-library-nvml NVML library]. For additional information see [https://slurm.schedmd.com/gres.html SLURM&amp;#039;s documentation].&lt;/div&gt;</summary>
		<author><name>GiulioFontana</name></author>
	</entry>
	<entry>
		<id>https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2564</id>
		<title>SLURM</title>
		<link rel="alternate" type="text/html" href="https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2564"/>
		<updated>2026-11-03T14:20:16Z</updated>

		<summary type="html">&lt;p&gt;GiulioFontana: /* Elements determining job priority */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page presents the features of SLURM that are most relevant to Mufasa&amp;#039;s [[Roles|Job Users]]. Job Users can submit jobs for execution, cancel their own jobs, and see other users&amp;#039; jobs (but not intervene on them).&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa &amp;#039;&amp;#039;&amp;#039;must use SLURM&amp;#039;&amp;#039;&amp;#039; to run resource-heavy processes, i.e. computing jobs that require one or more of the following:&lt;br /&gt;
* GPUs&lt;br /&gt;
* multiple CPUs&lt;br /&gt;
* powerful CPUs&lt;br /&gt;
* a significant amount of RAM&lt;br /&gt;
&lt;br /&gt;
In fact, only processes run via SLURM have access to all the resources of Mufasa. Processes run outside SLURM are executed by the [[System#Login server|login server]] virtual machine, which has minimal resources and no GPUs. Using SLURM is therefore the only way to execute resource-heavy jobs on Mufasa (this is a key difference between Mufasa 1.0 and Mufasa 2.0).&lt;br /&gt;
&lt;br /&gt;
= SLURM in a nutshell =&lt;br /&gt;
&lt;br /&gt;
Computation jobs on Mufasa needs to be launched via [[System#The SLURM job scheduling system|SLURM]]. SLURM provides jobs with access to the [[#System resources subjected to limitations|physical resources]] of Mufasa, such as CPUs, GPUs and RAM. Thanks to SLURM, processing jobs share system resources, optimising their occupation and availability. &lt;br /&gt;
&lt;br /&gt;
When a user runs a job, the job does not get executed immediately and is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039;. SLURM executes jobs according to their order in the queue: the top job in the queue gets executed as soon as the necessary resources are available, while jobs lower in the queue wait longer. The position of a job in the queue is due to the &amp;#039;&amp;#039;&amp;#039;[[#Job priority|priority]]&amp;#039;&amp;#039;&amp;#039; assigned to it by SLURM, with higher-priority jobs closer to the top. As a general rule:&lt;br /&gt;
&lt;br /&gt;
;: &amp;#039;&amp;#039;&amp;#039;the greater the fraction of Mufasa&amp;#039;s overall resources that a job asks for, the lower the job&amp;#039;s priority will be&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
The priority mechanism is used to encourage users to use Mufasa&amp;#039;s resources (i.e.: GPUs, CPUs, RAM, execution time) in an effective and equitable manner. This page includes a [[#How_to_maximise_the_priority_of_your_jobs|chart explaining how to maximise the priority of your jobs]].&lt;br /&gt;
&lt;br /&gt;
The &amp;#039;&amp;#039;&amp;#039;time&amp;#039;&amp;#039;&amp;#039; available to a job for its execution is controlled by SLURM. When a user requests execution of a job, they must specify the duration of the time slot that the job needs. The job must complete its execution before the end of the requested time slot, because at the end of such slot it gets killed by SLURM.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0 access to system resources is managed via SLURM&amp;#039;s [[#SLURM Quality of Service (QOS)|Quality of Service (QOS)]] mechanism (Mufasa 1.0 used [[#SLURM_partitions|partitions]] instead). While [[User Jobs#Running jobs with SLURM|launching a processing job via SLURM]], the user must always specify the chosen QOS. QOSes differ in the set of resources that they provide access to because each of them is designed to fit a given type of job.&lt;br /&gt;
&lt;br /&gt;
Mufasa sets limits to the number of jobs by the same user. This page includes a [[#Limits on jobs by the same user|table summarising such limits]].&lt;br /&gt;
&lt;br /&gt;
= SLURM Quality of Service (QOS) =&lt;br /&gt;
&lt;br /&gt;
Through &amp;#039;&amp;#039;&amp;#039;Quality of Services&amp;#039;&amp;#039;&amp;#039; (&amp;#039;&amp;#039;&amp;#039;QOSes&amp;#039;&amp;#039;&amp;#039;), SLURM lets system configurators assign a name to a set of related constraints.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, QOSes are used to define different levels of access to the server&amp;#039;s resources. When [[User Jobs|executing a job with SLURM]], a user must always &amp;#039;&amp;#039;&amp;#039;specify the QOS&amp;#039;&amp;#039;&amp;#039; that their job will use: this choice, in turn, determines what resources the job is able to access and influences the [[#Job priority|priority]] of the job.&lt;br /&gt;
&lt;br /&gt;
Mufasa&amp;#039;s QOSes and their features can be inspected with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list qos format=name%-11,priority,MaxSubmit,maxwall,maxtres%-80,mintres%-18&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following (see below for the same data in table form):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
Name          Priority MaxSubmit     MaxWall MaxTRES                                                                          MinTRES            &lt;br /&gt;
----------- ---------- --------- ----------- -------------------------------------------------------------------------------- ------------------ &lt;br /&gt;
normal               0                                                                                                                           &lt;br /&gt;
nogpu                4         1  3-00:00:00 cpu=16,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=128G                    &lt;br /&gt;
gpuheavy-20          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=2,mem=128G             gres/gpu:4g.20gb=1 &lt;br /&gt;
gpuheavy-40          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=1,gres/gpu:4g.20gb=0,mem=128G             gres/gpu:40gb=1    &lt;br /&gt;
gpulight             8         1    12:00:00 cpu=2,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpu                  2         1  1-00:00:00 cpu=8,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpuwide              2         2  1-00:00:00 cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=1,mem=64G              gres/gpu:4g.20gb=1 &lt;br /&gt;
build               32         1    02:00:00 cpu=2,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=16G&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The columns of this output are the following:&lt;br /&gt;
&lt;br /&gt;
:; Name&lt;br /&gt;
:: name of the QOS&lt;br /&gt;
&lt;br /&gt;
:; Priority&lt;br /&gt;
:: priority tier associated to the QOS (higher value = higher priority): see [[#Job priority|Job priority]] for details&lt;br /&gt;
&lt;br /&gt;
:; MaxSubmit&lt;br /&gt;
:: maximum number of jobs from a single user that can be submitted to SLURM with this QOS; submitted jobs include both running and queued jobs&lt;br /&gt;
:: See [[#Limits on jobs by the same user|Limits on jobs by the same user]] for an overview of the limits on jobs set by Mufasa.&lt;br /&gt;
&lt;br /&gt;
:; MaxWall&lt;br /&gt;
:: maximum wall clock duration of the jobs using the QOS (after which they are killed by SLURM), in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
:: For some QOSes these are not set: it means that they are determined by the [[#SLURM partitions|partition]]. Partitions also define the [[#Default values|default duration]] of jobs.&lt;br /&gt;
&lt;br /&gt;
:; MaxTRES&lt;br /&gt;
:: amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) available to a job using the QOS, where&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;cpu=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of CPUs (i.e., processor cores) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of CPUs specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;gres/&amp;#039;&amp;#039;gpu:Type&amp;#039;&amp;#039;=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of GPUs of class &amp;lt;code&amp;gt;&amp;#039;&amp;#039;Type&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; (see [[User Jobs#gres syntax|&amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax]]) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; (for QOSes that allow access to GPUs) if not specified, the job cannot be launched&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;mem=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;G&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum amount of system RAM is &amp;#039;&amp;#039;K&amp;#039;&amp;#039; GBytes&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of RAM specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
&lt;br /&gt;
:; MinTRES&lt;br /&gt;
:: minimum amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) that a job using the QOS must request in order to actually get executed by SLURM.&lt;br /&gt;
:: (If your job does not actually need these resources, you&amp;#039;ve chosen the wrong QOS: you can use one with a higher priority.)&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; QOS is the default one, and exists only to ensure that users always specify a QOS when running a job. Since &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; has zero priority and no resources, a job run using this QOS would never be run.&lt;br /&gt;
&lt;br /&gt;
The information provided by the &amp;lt;code&amp;gt;sacctmgr list qos&amp;lt;/code&amp;gt; command above is summarised by the following table:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! QOS&lt;br /&gt;
! Priority&amp;lt;/br&amp;gt;tier&lt;br /&gt;
! Max&amp;lt;/br&amp;gt;Submit&lt;br /&gt;
! MaxWall&amp;lt;/br&amp;gt;[h]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;CPUs&lt;br /&gt;
! max RAM&amp;lt;/br&amp;gt;[GB]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;3g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;4g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;40GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | build&lt;br /&gt;
| 32&lt;br /&gt;
| 1&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 16&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpulight&lt;br /&gt;
| 8&lt;br /&gt;
| 1&lt;br /&gt;
| 12&lt;br /&gt;
| 2&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | nogpu&lt;br /&gt;
| 4&lt;br /&gt;
| 1&lt;br /&gt;
| 72&lt;br /&gt;
| 16&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpu&lt;br /&gt;
| 2&lt;br /&gt;
| 1&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuwide&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-20&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| 2 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-40&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
A key piece of information in the table above is the &amp;#039;&amp;#039;&amp;#039;priority tier&amp;#039;&amp;#039;&amp;#039; associated to each QOS. In Mufasa 2.0, priority tiers are used to encourage users to use the &amp;#039;&amp;#039;&amp;#039;least powerful QOS that is compatible with their needs&amp;#039;&amp;#039;&amp;#039;, where &amp;quot;powerful&amp;quot; means &amp;quot;rich with resources&amp;quot;. Less powerful QOSes increase the priority of the jobs that use them, so these jobs tend to be executed sooner. See [[#Job priority|Job priority]] for details about how priority affects the execution order of jobs in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
== The &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS ==&lt;br /&gt;
&lt;br /&gt;
This QOS is specifically designed to be used by Mufasa users to &amp;#039;&amp;#039;&amp;#039;quickly build [[System#Containers|container images]]&amp;#039;&amp;#039;&amp;#039;. Its associated priority tier is very high: this, combined with the fact that jobs using this QOS require few resources, means that such jobs usually get executed very soon.&lt;br /&gt;
&lt;br /&gt;
On the other hand, the limited resources and the lack of access to the GPUs make the &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS unsuitable for other tasks.&lt;br /&gt;
&lt;br /&gt;
See [[Singularity#Building Singularity images|Building Singularity images]] for directions about building Singularity container images.&lt;br /&gt;
&lt;br /&gt;
== Restricted QOSes ==&lt;br /&gt;
&lt;br /&gt;
In Mufasa, the most powerful QOSes are reserved to researchers (called &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users: these include academic personnel and Ph.D. students). M.Sc. students, called &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users, cannot use them while running jobs.&lt;br /&gt;
&lt;br /&gt;
See [[#research users and students users|below]] to understand how &amp;#039;&amp;#039;user categories&amp;#039;&amp;#039; work in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
= &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users =&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa belong to two &amp;#039;&amp;#039;&amp;#039;user categories&amp;#039;&amp;#039;&amp;#039;. Categories provide users with different access to  to Mufasa&amp;#039;s resources: the idea being to provide researchers with more access while still letting students benefit from the server&amp;#039;s capabilities.&lt;br /&gt;
&lt;br /&gt;
User categories in Mufasa are the following:&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. academic personnel and Ph.D. students&lt;br /&gt;
::: * can use all QOSes, including the the [[#Restricted QOSes|restricted ones]]&lt;br /&gt;
::: * their jobs have a higher &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a higher number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. M.Sc. students&lt;br /&gt;
::: * cannot use the [[#Restricted QOSes|restricted QOSes]]&lt;br /&gt;
::: * their jobs have a lower &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a lower number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
You can inspect the differences in priority and running jobs between &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association format=&amp;quot;account,priority,maxjobs&amp;quot; | grep -E &amp;#039;Account|research|students&amp;#039;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
   Account   Priority MaxJobs &lt;br /&gt;
  research          4       2 &lt;br /&gt;
  students          1       1&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
To know what limits apply to your own user, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association where user=$USER format=&amp;quot;user,priority,maxjobs,qos%-60&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
      User   Priority MaxJobs QOS                                                          &lt;br /&gt;
---------- ---------- ------- ------------------------------------------------------------ &lt;br /&gt;
    preali          4       2 build,gpu,gpuheavy-20,gpuheavy-40,gpulight,gpuwide,nogpu&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The list under &amp;quot;QOS&amp;quot; shows what QOSes your user is allowed to use when [[User Jobs|running jobs]]. &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users can use all of them, while &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users can only access a subset of them.&lt;br /&gt;
&lt;br /&gt;
= Limits on jobs by the same user =&lt;br /&gt;
&lt;br /&gt;
Mufasa enforces limits on the number of jobs from a single user. Such limits aim at preventing users from &amp;quot;hogging&amp;quot; system resources, and apply to:&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;submitted jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that the user asked SLURM to execute, each of which may currently be either running or queued&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;running jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that are currently in execution&lt;br /&gt;
&lt;br /&gt;
Limits on submitted jobs are set via [[#SLURM Quality of Service (QOS)|QOSes]], while limits on running jobs are set via [[#research users and students users|user category]].&lt;br /&gt;
&lt;br /&gt;
The following table summarises what limits exist:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! &lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;running&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;submitted&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | global limits&amp;lt;br/&amp;gt;(system-wide)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to limits set by individual QOSes (below)&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | limits for&amp;lt;br/&amp;gt;individual&amp;lt;/br&amp;gt;QOSes&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to global limits (above)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for the&amp;#039;&amp;#039; &amp;lt;code&amp;gt;gpuwide&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;QOS&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for each of the other QOSes&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
= Job priority =&lt;br /&gt;
&lt;br /&gt;
Once the execution of a job has been requested, the job is not run immediately: it is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039; by SLURM, together with all the other jobs awaiting execution. The job on top of the queue at any time is the first to be put into execution as soon as the resources it requires are available. &lt;br /&gt;
&lt;br /&gt;
The order of the items in the job queue depends on their &amp;#039;&amp;#039;&amp;#039;priority&amp;#039;&amp;#039;&amp;#039;. The job with the highest priority is put by SLURM on top of the queue, while all other jobs are put in the queue in order of descending priority.&lt;br /&gt;
&lt;br /&gt;
The goal of SLURM is to maximise resource availability: i.e., to ensure the shorter possible wait time before job execution. To achieve this goal, in Mufasa SLURM is configured to &amp;#039;&amp;#039;&amp;#039;encourage users not to ask for resources or execution time that their job doesn&amp;#039;t need&amp;#039;&amp;#039;&amp;#039;. This is done via the priority mechanism: the more resources and/or time a job requests, the lower its priority will be; and the later it will be executed.&lt;br /&gt;
&lt;br /&gt;
Priority management in Mufasa is designed to set up a &amp;#039;&amp;#039;&amp;#039;virtuous cycle&amp;#039;&amp;#039;&amp;#039; where users, by carefully choosing what to ask for, obtain two results:&lt;br /&gt;
* they ensure that their job is executed as soon as possible;&lt;br /&gt;
* they leave as much as possible of Mufasa&amp;#039;s resources free for other users&amp;#039;s jobs.&lt;br /&gt;
&lt;br /&gt;
== Elements determining job priority ==&lt;br /&gt;
In Mufasa, the priority of a job is computed by SLURM according to the following elements:&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#research users and students users|User category]]&amp;#039;&amp;#039;&amp;#039; (i.e., &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;)&lt;br /&gt;
::: Used to provide higher priority to jobs run by &amp;#039;&amp;#039;&amp;#039;research personnel&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#SLURM Quality of Service (QOS)|QOS]]&amp;#039;&amp;#039;&amp;#039; used by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;less resources&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Number of CPUs&amp;#039;&amp;#039;&amp;#039; requested by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;a lower number of CPUs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job Age&amp;#039;&amp;#039;&amp;#039;, i.e. the time that the job has been waiting in the queue&lt;br /&gt;
::: Used to provide higher priority to jobs which have been &amp;#039;&amp;#039;&amp;#039;queued for a longer time&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;FairShare&amp;#039;&amp;#039;&amp;#039;, i.e. a factor computed by SLURM to balance use of the system by different users&lt;br /&gt;
::: Used to provide higher priority to jobs by users who &amp;#039;&amp;#039;&amp;#039;used Mufasa&amp;#039;s resources less than others&amp;#039;&amp;#039;&amp;#039; (see [[#How FairShare works|below]])&lt;br /&gt;
&lt;br /&gt;
You can understand how the priority of your jobs depends on each of these factors using command&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sprio&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
A tailored version of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; that only shows the parameters listed above is the following:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sprio -o &amp;quot;%6i %9u %.5Y %.5A %.5B %.5F %.5Q%.10T&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
Here is an example of its output:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
JOBID  USER      PRIOR   AGE ASSOC FAIRS   QOS      TRES&lt;br /&gt;
5774   itombini   3485   899  2500   136   750  cpu=-800&lt;br /&gt;
5780   fbottazzi  2259   413   625  1071   750  cpu=-600&lt;br /&gt;
5783   mzoccola   1759   210   625   974   750  cpu=-800&lt;br /&gt;
5790   amoglia    2710   149  2500   487   375  cpu=-800&lt;br /&gt;
5791   qliu       2225   131  2500    19   375  cpu=-800&lt;br /&gt;
5792   fpastori   2190    77  2500    39   375  cpu=-800&lt;br /&gt;
5793   bagostini  2027    11   625  1091  1500 cpu=-1200&lt;br /&gt;
5794   lpassoni   2510     2  2500    58   750  cpu=-800&lt;br /&gt;
5795   lpassoni   2510     2  2500    58   750  cpu=-800&lt;br /&gt;
5796   adurat     3094     2  2500   643   750  cpu=-800&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
In it, column ASSOC (i.e., association) corresponds to user category and column TRES corresponds to the number of CPUs. Column PRIOR provides the overall priority of each job.&lt;br /&gt;
&lt;br /&gt;
=== How FairShare works ===&lt;br /&gt;
In Mufasa 2.0, the FairShare of each user is computed according to these rules:&lt;br /&gt;
# It considers the quantity used by (current and past) user jobs of:&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;CPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of CPUs used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;RAM&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the amount of RAM used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;GPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of GPUs used&lt;br /&gt;
#** note: 40 GB GPUs have double the impact on FairShare than 20 GB GPUs&lt;br /&gt;
# The impact on FairShare of any use of resources is &amp;#039;&amp;#039;&amp;#039;proportional to its duration&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
#* example: using 32 GB of RAM for 48h has the same impact of using 64 GB of RAM for 24h&lt;br /&gt;
# FairShare has a &amp;quot;fading memory&amp;quot;: i.e., resource use has less and less impact on FairShare the farther in the past it is&lt;br /&gt;
#* consequence: over time, the &amp;quot;history&amp;quot; of a user gets forgotten by FairShare&lt;br /&gt;
&lt;br /&gt;
== How to maximise the priority of your jobs ==&lt;br /&gt;
&lt;br /&gt;
Every time you run a SLURM job, follow these guidelines:&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Choose the less powerful QOS compatible with the needs of your job&lt;br /&gt;
:: QOSes with access to less resources have higher base priority&lt;br /&gt;
&lt;br /&gt;
; Only request CPUs and RAM that your job really needs&lt;br /&gt;
:: Usually, code exploits multiple CPUs only if specifically designed to do so: if unsure, only ask for 1 CPU&lt;br /&gt;
:: The fewer resources your job asks for, the less it will wait before they become available&lt;br /&gt;
:: Asking for fewer CPUs and/or less RAM improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Do not request more time than your jobs needs to complete&lt;br /&gt;
:: Make a worst-case estimate and only ask for that duration&lt;br /&gt;
:: Asking for less of Mufasa&amp;#039;s time improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Debug and test code using less powerful QOSes before running it with more powerful QOSes&lt;br /&gt;
:: Your test jobs will get a higher priority and your FairShare will improve&lt;br /&gt;
&lt;br /&gt;
; Cancel jobs that you don&amp;#039;t need them anymore&lt;br /&gt;
:: Always use [[User_Jobs#Cancelling_a_job_with_scancel|scancel]] to delete completed (or crashed) jobs: your Fairshare will improve&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Protip: it&amp;#039;s a good idea to [[User Jobs#Looking for unused GPUs|check for unused GPUs]] before choosing what to request. Requesting a GPU that is currently idle will help your job get executed sooner.&lt;br /&gt;
&lt;br /&gt;
= SLURM partitions =&lt;br /&gt;
&lt;br /&gt;
Partitions are another mechanism provided by SLURM to create different levels of access to system resources. Since in Mufasa 2.0 access to resources is controlled via [[#SLURM Quality of Service (QOS)|QOSes]], partitions are not very relevant. (This is one of the main differences between Mufasa 1.0 and Mufasa 2.0.)&lt;br /&gt;
&lt;br /&gt;
Note, however, that the default values for some features of SLURM jobs (e.g., duration) are [[#Default values|set by the partition]].&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, there is a single SLURM partition, called &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt;, and all jobs run on it. The state of &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt; can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sinfo -o &amp;quot;%10P %5a %9T %11L %10l&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PARTITION  AVAIL STATE     DEFAULTTIME TIMELIMIT &lt;br /&gt;
jobs*      up    idle      1:00:00     3-00:00:00&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where columns correspond to the following information:&lt;br /&gt;
&lt;br /&gt;
:; PARTITION&lt;br /&gt;
:: name of the partition; the asterisks indicates that it&amp;#039;s the default one&lt;br /&gt;
&lt;br /&gt;
:; AVAIL&lt;br /&gt;
:: state/availability of the partition: see [[#Partition availability|below]]&lt;br /&gt;
&lt;br /&gt;
:; STATE&lt;br /&gt;
:: state (using [https://slurm.schedmd.com/sinfo.html#SECTION_NODE-STATE-CODES these codes])&lt;br /&gt;
:: typical values are &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;mixed&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that some of the resources are busy executing jobs while other are idle, and &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;allocated&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that all of the resources are in use&lt;br /&gt;
&lt;br /&gt;
:; DEFAULTTIME&lt;br /&gt;
:: default runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:; TIMELIMIT&lt;br /&gt;
:: maximum runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
The asterisk at the end of the partition name indicates the default partition, i.e. the one on which jobs which do not ask for a specific partition are run.&lt;br /&gt;
&lt;br /&gt;
Command &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; does not tell you about the &amp;#039;&amp;#039;jobs&amp;#039;&amp;#039; submitted to a partition. This information is obtained, instead, with [[User Jobs#Inspecting jobs with squeue|command &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;]].&lt;br /&gt;
&lt;br /&gt;
== Partition availability ==&lt;br /&gt;
&lt;br /&gt;
The most important information that &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; provides is the &amp;#039;&amp;#039;&amp;#039;availability&amp;#039;&amp;#039;&amp;#039; (also called &amp;#039;&amp;#039;&amp;#039;state&amp;#039;&amp;#039;&amp;#039;) of partitions. This is shown in column &amp;quot;AVAIL&amp;quot;. Possible partition states are:&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is available&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed as soon as resources allow&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is in the process of becoming unavailable (i.e., of entering the &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; state: see below)&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is unavailable&lt;br /&gt;
:: There are no running jobs&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
When a partition goes from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; no harm is done to running jobs. In a normally functioning SLURM system, the passage from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; happens only when no jobs are running on the partition. If (e.g., due to a malfunction) the passage happens with jobs still running, they get killed.&lt;br /&gt;
&lt;br /&gt;
A partition in state &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; requires intervention by a [[Roles|Job Administrator]] to be restored to &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
== Default values ==&lt;br /&gt;
&lt;br /&gt;
The features of SLURM partitions, including the &amp;#039;&amp;#039;&amp;#039;default values&amp;#039;&amp;#039;&amp;#039; which are applied to jobs that do not make explicit requests, can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scontrol show partition&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PartitionName=jobs&lt;br /&gt;
   AllowGroups=ALL AllowAccounts=ALL AllowQos=nogpu,gpulight,gpu,gpuwide,gpuheavy-20,gpuheavy-40&lt;br /&gt;
   AllocNodes=ALL Default=YES QoS=N/A&lt;br /&gt;
=&amp;gt; DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO&lt;br /&gt;
   MaxNodes=UNLIMITED MaxTime=3-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED&lt;br /&gt;
   Nodes=gn01&lt;br /&gt;
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO&lt;br /&gt;
   OverTimeLimit=NONE PreemptMode=OFF&lt;br /&gt;
   State=UP TotalCPUs=48 TotalNodes=1 SelectTypeParameters=NONE&lt;br /&gt;
   JobDefaults=(null)&lt;br /&gt;
=&amp;gt; DefMemPerNode=4096 MaxMemPerNode=UNLIMITED&lt;br /&gt;
   TRES=cpu=48,mem=1011435M,node=1,billing=49,gres/gpu=13,gres/gpu:3g.20gb=5,gres/gpu:40gb=3,gres/gpu:4g.20gb=5&lt;br /&gt;
   TRESBillingWeights=cpu=1.0,gres/gpu:3g.20gb=6.0,gres/gpu:4g.20gb=6.0,gres/gpu:40gb=6.0,mem=0.05g&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In the example, we have highlighted with  &amp;#039;&amp;#039;&amp;#039;=&amp;gt;&amp;#039;&amp;#039;&amp;#039;  the most relevant default values for Mufasa users, i.e.:&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefaultTime&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default execution time assigned to a job run on the partition (e.g., 1 hour)&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefMemPerNode&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default amount of RAM assigned to a job run on the partition (e.g., 4GB)&lt;br /&gt;
&lt;br /&gt;
= System resources subjected to limitations =&lt;br /&gt;
&lt;br /&gt;
In systems based on SLURM like Mufasa, &amp;#039;&amp;#039;&amp;#039;TRES (Trackable RESources)&amp;#039;&amp;#039;&amp;#039; are (from [https://slurm.schedmd.com/tres.html SLURM&amp;#039;s documentation] &amp;quot;&amp;#039;&amp;#039;resources that can be tracked for usage or used to enforce limits against.&amp;#039;&amp;#039;&amp;quot;&lt;br /&gt;
&lt;br /&gt;
TRES include CPUs, RAM and &amp;#039;&amp;#039;&amp;#039;GRES&amp;#039;&amp;#039;&amp;#039;. The last term stands for &amp;#039;&amp;#039;Generic RESources&amp;#039;&amp;#039; that a job may need for its execution. In Mufasa, the only &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; resources are the GPUs.&lt;br /&gt;
&lt;br /&gt;
== &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax ==&lt;br /&gt;
&lt;br /&gt;
To ask SLURM to assign GRES resources (i.e., GPUs) to a job, a special syntax must be used. Precisely, the name of each GPU resource takes the form&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:Name:Type&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
Considering the [[System#CPUs and GPUs|GPU complement of Mufasa]], &amp;lt;code&amp;gt;Type&amp;lt;/code&amp;gt; takes the following values:&lt;br /&gt;
&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:40gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 40 Gbytes of RAM&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:4g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 4 compute units&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 3 compute units&lt;br /&gt;
&lt;br /&gt;
So, for instance,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
identifies a resource corresponding to a GPU with 20 GB of RAM and 3 compute units.&lt;br /&gt;
&lt;br /&gt;
When asking for a GRES resource (e.g., in an &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command or an &amp;lt;code&amp;gt;SBATCH&amp;lt;/code&amp;gt; directive of an [[User Jobs#Using execution scripts to run jobs|execution script]]), the syntax required by SLURM is&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:&amp;lt;Type&amp;gt;:&amp;lt;Quantity&amp;gt;&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
where &amp;lt;code&amp;gt;Quantity&amp;lt;/code&amp;gt; is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of type &amp;lt;code&amp;gt;4g.20gb&amp;lt;/code&amp;gt; the syntax is&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:4g.20gb:2&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SLURM&amp;#039;s &amp;#039;&amp;#039;generic resources&amp;#039;&amp;#039; are defined in &amp;lt;code&amp;gt;/etc/slurm/gres.conf&amp;lt;/code&amp;gt;. In order to make GPUs available to SLURM&amp;#039;s &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; management, Mufasa makes use of Nvidia&amp;#039;s [https://developer.nvidia.com/nvidia-management-library-nvml NVML library]. For additional information see [https://slurm.schedmd.com/gres.html SLURM&amp;#039;s documentation].&lt;/div&gt;</summary>
		<author><name>GiulioFontana</name></author>
	</entry>
	<entry>
		<id>https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2563</id>
		<title>SLURM</title>
		<link rel="alternate" type="text/html" href="https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2563"/>
		<updated>2026-11-03T14:19:40Z</updated>

		<summary type="html">&lt;p&gt;GiulioFontana: /* Elements determining job priority */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page presents the features of SLURM that are most relevant to Mufasa&amp;#039;s [[Roles|Job Users]]. Job Users can submit jobs for execution, cancel their own jobs, and see other users&amp;#039; jobs (but not intervene on them).&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa &amp;#039;&amp;#039;&amp;#039;must use SLURM&amp;#039;&amp;#039;&amp;#039; to run resource-heavy processes, i.e. computing jobs that require one or more of the following:&lt;br /&gt;
* GPUs&lt;br /&gt;
* multiple CPUs&lt;br /&gt;
* powerful CPUs&lt;br /&gt;
* a significant amount of RAM&lt;br /&gt;
&lt;br /&gt;
In fact, only processes run via SLURM have access to all the resources of Mufasa. Processes run outside SLURM are executed by the [[System#Login server|login server]] virtual machine, which has minimal resources and no GPUs. Using SLURM is therefore the only way to execute resource-heavy jobs on Mufasa (this is a key difference between Mufasa 1.0 and Mufasa 2.0).&lt;br /&gt;
&lt;br /&gt;
= SLURM in a nutshell =&lt;br /&gt;
&lt;br /&gt;
Computation jobs on Mufasa needs to be launched via [[System#The SLURM job scheduling system|SLURM]]. SLURM provides jobs with access to the [[#System resources subjected to limitations|physical resources]] of Mufasa, such as CPUs, GPUs and RAM. Thanks to SLURM, processing jobs share system resources, optimising their occupation and availability. &lt;br /&gt;
&lt;br /&gt;
When a user runs a job, the job does not get executed immediately and is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039;. SLURM executes jobs according to their order in the queue: the top job in the queue gets executed as soon as the necessary resources are available, while jobs lower in the queue wait longer. The position of a job in the queue is due to the &amp;#039;&amp;#039;&amp;#039;[[#Job priority|priority]]&amp;#039;&amp;#039;&amp;#039; assigned to it by SLURM, with higher-priority jobs closer to the top. As a general rule:&lt;br /&gt;
&lt;br /&gt;
;: &amp;#039;&amp;#039;&amp;#039;the greater the fraction of Mufasa&amp;#039;s overall resources that a job asks for, the lower the job&amp;#039;s priority will be&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
The priority mechanism is used to encourage users to use Mufasa&amp;#039;s resources (i.e.: GPUs, CPUs, RAM, execution time) in an effective and equitable manner. This page includes a [[#How_to_maximise_the_priority_of_your_jobs|chart explaining how to maximise the priority of your jobs]].&lt;br /&gt;
&lt;br /&gt;
The &amp;#039;&amp;#039;&amp;#039;time&amp;#039;&amp;#039;&amp;#039; available to a job for its execution is controlled by SLURM. When a user requests execution of a job, they must specify the duration of the time slot that the job needs. The job must complete its execution before the end of the requested time slot, because at the end of such slot it gets killed by SLURM.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0 access to system resources is managed via SLURM&amp;#039;s [[#SLURM Quality of Service (QOS)|Quality of Service (QOS)]] mechanism (Mufasa 1.0 used [[#SLURM_partitions|partitions]] instead). While [[User Jobs#Running jobs with SLURM|launching a processing job via SLURM]], the user must always specify the chosen QOS. QOSes differ in the set of resources that they provide access to because each of them is designed to fit a given type of job.&lt;br /&gt;
&lt;br /&gt;
Mufasa sets limits to the number of jobs by the same user. This page includes a [[#Limits on jobs by the same user|table summarising such limits]].&lt;br /&gt;
&lt;br /&gt;
= SLURM Quality of Service (QOS) =&lt;br /&gt;
&lt;br /&gt;
Through &amp;#039;&amp;#039;&amp;#039;Quality of Services&amp;#039;&amp;#039;&amp;#039; (&amp;#039;&amp;#039;&amp;#039;QOSes&amp;#039;&amp;#039;&amp;#039;), SLURM lets system configurators assign a name to a set of related constraints.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, QOSes are used to define different levels of access to the server&amp;#039;s resources. When [[User Jobs|executing a job with SLURM]], a user must always &amp;#039;&amp;#039;&amp;#039;specify the QOS&amp;#039;&amp;#039;&amp;#039; that their job will use: this choice, in turn, determines what resources the job is able to access and influences the [[#Job priority|priority]] of the job.&lt;br /&gt;
&lt;br /&gt;
Mufasa&amp;#039;s QOSes and their features can be inspected with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list qos format=name%-11,priority,MaxSubmit,maxwall,maxtres%-80,mintres%-18&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following (see below for the same data in table form):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
Name          Priority MaxSubmit     MaxWall MaxTRES                                                                          MinTRES            &lt;br /&gt;
----------- ---------- --------- ----------- -------------------------------------------------------------------------------- ------------------ &lt;br /&gt;
normal               0                                                                                                                           &lt;br /&gt;
nogpu                4         1  3-00:00:00 cpu=16,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=128G                    &lt;br /&gt;
gpuheavy-20          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=2,mem=128G             gres/gpu:4g.20gb=1 &lt;br /&gt;
gpuheavy-40          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=1,gres/gpu:4g.20gb=0,mem=128G             gres/gpu:40gb=1    &lt;br /&gt;
gpulight             8         1    12:00:00 cpu=2,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpu                  2         1  1-00:00:00 cpu=8,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpuwide              2         2  1-00:00:00 cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=1,mem=64G              gres/gpu:4g.20gb=1 &lt;br /&gt;
build               32         1    02:00:00 cpu=2,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=16G&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The columns of this output are the following:&lt;br /&gt;
&lt;br /&gt;
:; Name&lt;br /&gt;
:: name of the QOS&lt;br /&gt;
&lt;br /&gt;
:; Priority&lt;br /&gt;
:: priority tier associated to the QOS (higher value = higher priority): see [[#Job priority|Job priority]] for details&lt;br /&gt;
&lt;br /&gt;
:; MaxSubmit&lt;br /&gt;
:: maximum number of jobs from a single user that can be submitted to SLURM with this QOS; submitted jobs include both running and queued jobs&lt;br /&gt;
:: See [[#Limits on jobs by the same user|Limits on jobs by the same user]] for an overview of the limits on jobs set by Mufasa.&lt;br /&gt;
&lt;br /&gt;
:; MaxWall&lt;br /&gt;
:: maximum wall clock duration of the jobs using the QOS (after which they are killed by SLURM), in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
:: For some QOSes these are not set: it means that they are determined by the [[#SLURM partitions|partition]]. Partitions also define the [[#Default values|default duration]] of jobs.&lt;br /&gt;
&lt;br /&gt;
:; MaxTRES&lt;br /&gt;
:: amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) available to a job using the QOS, where&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;cpu=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of CPUs (i.e., processor cores) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of CPUs specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;gres/&amp;#039;&amp;#039;gpu:Type&amp;#039;&amp;#039;=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of GPUs of class &amp;lt;code&amp;gt;&amp;#039;&amp;#039;Type&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; (see [[User Jobs#gres syntax|&amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax]]) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; (for QOSes that allow access to GPUs) if not specified, the job cannot be launched&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;mem=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;G&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum amount of system RAM is &amp;#039;&amp;#039;K&amp;#039;&amp;#039; GBytes&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of RAM specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
&lt;br /&gt;
:; MinTRES&lt;br /&gt;
:: minimum amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) that a job using the QOS must request in order to actually get executed by SLURM.&lt;br /&gt;
:: (If your job does not actually need these resources, you&amp;#039;ve chosen the wrong QOS: you can use one with a higher priority.)&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; QOS is the default one, and exists only to ensure that users always specify a QOS when running a job. Since &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; has zero priority and no resources, a job run using this QOS would never be run.&lt;br /&gt;
&lt;br /&gt;
The information provided by the &amp;lt;code&amp;gt;sacctmgr list qos&amp;lt;/code&amp;gt; command above is summarised by the following table:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! QOS&lt;br /&gt;
! Priority&amp;lt;/br&amp;gt;tier&lt;br /&gt;
! Max&amp;lt;/br&amp;gt;Submit&lt;br /&gt;
! MaxWall&amp;lt;/br&amp;gt;[h]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;CPUs&lt;br /&gt;
! max RAM&amp;lt;/br&amp;gt;[GB]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;3g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;4g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;40GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | build&lt;br /&gt;
| 32&lt;br /&gt;
| 1&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 16&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpulight&lt;br /&gt;
| 8&lt;br /&gt;
| 1&lt;br /&gt;
| 12&lt;br /&gt;
| 2&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | nogpu&lt;br /&gt;
| 4&lt;br /&gt;
| 1&lt;br /&gt;
| 72&lt;br /&gt;
| 16&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpu&lt;br /&gt;
| 2&lt;br /&gt;
| 1&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuwide&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-20&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| 2 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-40&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
A key piece of information in the table above is the &amp;#039;&amp;#039;&amp;#039;priority tier&amp;#039;&amp;#039;&amp;#039; associated to each QOS. In Mufasa 2.0, priority tiers are used to encourage users to use the &amp;#039;&amp;#039;&amp;#039;least powerful QOS that is compatible with their needs&amp;#039;&amp;#039;&amp;#039;, where &amp;quot;powerful&amp;quot; means &amp;quot;rich with resources&amp;quot;. Less powerful QOSes increase the priority of the jobs that use them, so these jobs tend to be executed sooner. See [[#Job priority|Job priority]] for details about how priority affects the execution order of jobs in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
== The &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS ==&lt;br /&gt;
&lt;br /&gt;
This QOS is specifically designed to be used by Mufasa users to &amp;#039;&amp;#039;&amp;#039;quickly build [[System#Containers|container images]]&amp;#039;&amp;#039;&amp;#039;. Its associated priority tier is very high: this, combined with the fact that jobs using this QOS require few resources, means that such jobs usually get executed very soon.&lt;br /&gt;
&lt;br /&gt;
On the other hand, the limited resources and the lack of access to the GPUs make the &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS unsuitable for other tasks.&lt;br /&gt;
&lt;br /&gt;
See [[Singularity#Building Singularity images|Building Singularity images]] for directions about building Singularity container images.&lt;br /&gt;
&lt;br /&gt;
== Restricted QOSes ==&lt;br /&gt;
&lt;br /&gt;
In Mufasa, the most powerful QOSes are reserved to researchers (called &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users: these include academic personnel and Ph.D. students). M.Sc. students, called &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users, cannot use them while running jobs.&lt;br /&gt;
&lt;br /&gt;
See [[#research users and students users|below]] to understand how &amp;#039;&amp;#039;user categories&amp;#039;&amp;#039; work in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
= &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users =&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa belong to two &amp;#039;&amp;#039;&amp;#039;user categories&amp;#039;&amp;#039;&amp;#039;. Categories provide users with different access to  to Mufasa&amp;#039;s resources: the idea being to provide researchers with more access while still letting students benefit from the server&amp;#039;s capabilities.&lt;br /&gt;
&lt;br /&gt;
User categories in Mufasa are the following:&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. academic personnel and Ph.D. students&lt;br /&gt;
::: * can use all QOSes, including the the [[#Restricted QOSes|restricted ones]]&lt;br /&gt;
::: * their jobs have a higher &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a higher number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. M.Sc. students&lt;br /&gt;
::: * cannot use the [[#Restricted QOSes|restricted QOSes]]&lt;br /&gt;
::: * their jobs have a lower &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a lower number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
You can inspect the differences in priority and running jobs between &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association format=&amp;quot;account,priority,maxjobs&amp;quot; | grep -E &amp;#039;Account|research|students&amp;#039;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
   Account   Priority MaxJobs &lt;br /&gt;
  research          4       2 &lt;br /&gt;
  students          1       1&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
To know what limits apply to your own user, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association where user=$USER format=&amp;quot;user,priority,maxjobs,qos%-60&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
      User   Priority MaxJobs QOS                                                          &lt;br /&gt;
---------- ---------- ------- ------------------------------------------------------------ &lt;br /&gt;
    preali          4       2 build,gpu,gpuheavy-20,gpuheavy-40,gpulight,gpuwide,nogpu&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The list under &amp;quot;QOS&amp;quot; shows what QOSes your user is allowed to use when [[User Jobs|running jobs]]. &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users can use all of them, while &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users can only access a subset of them.&lt;br /&gt;
&lt;br /&gt;
= Limits on jobs by the same user =&lt;br /&gt;
&lt;br /&gt;
Mufasa enforces limits on the number of jobs from a single user. Such limits aim at preventing users from &amp;quot;hogging&amp;quot; system resources, and apply to:&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;submitted jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that the user asked SLURM to execute, each of which may currently be either running or queued&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;running jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that are currently in execution&lt;br /&gt;
&lt;br /&gt;
Limits on submitted jobs are set via [[#SLURM Quality of Service (QOS)|QOSes]], while limits on running jobs are set via [[#research users and students users|user category]].&lt;br /&gt;
&lt;br /&gt;
The following table summarises what limits exist:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! &lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;running&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;submitted&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | global limits&amp;lt;br/&amp;gt;(system-wide)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to limits set by individual QOSes (below)&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | limits for&amp;lt;br/&amp;gt;individual&amp;lt;/br&amp;gt;QOSes&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to global limits (above)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for the&amp;#039;&amp;#039; &amp;lt;code&amp;gt;gpuwide&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;QOS&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for each of the other QOSes&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
= Job priority =&lt;br /&gt;
&lt;br /&gt;
Once the execution of a job has been requested, the job is not run immediately: it is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039; by SLURM, together with all the other jobs awaiting execution. The job on top of the queue at any time is the first to be put into execution as soon as the resources it requires are available. &lt;br /&gt;
&lt;br /&gt;
The order of the items in the job queue depends on their &amp;#039;&amp;#039;&amp;#039;priority&amp;#039;&amp;#039;&amp;#039;. The job with the highest priority is put by SLURM on top of the queue, while all other jobs are put in the queue in order of descending priority.&lt;br /&gt;
&lt;br /&gt;
The goal of SLURM is to maximise resource availability: i.e., to ensure the shorter possible wait time before job execution. To achieve this goal, in Mufasa SLURM is configured to &amp;#039;&amp;#039;&amp;#039;encourage users not to ask for resources or execution time that their job doesn&amp;#039;t need&amp;#039;&amp;#039;&amp;#039;. This is done via the priority mechanism: the more resources and/or time a job requests, the lower its priority will be; and the later it will be executed.&lt;br /&gt;
&lt;br /&gt;
Priority management in Mufasa is designed to set up a &amp;#039;&amp;#039;&amp;#039;virtuous cycle&amp;#039;&amp;#039;&amp;#039; where users, by carefully choosing what to ask for, obtain two results:&lt;br /&gt;
* they ensure that their job is executed as soon as possible;&lt;br /&gt;
* they leave as much as possible of Mufasa&amp;#039;s resources free for other users&amp;#039;s jobs.&lt;br /&gt;
&lt;br /&gt;
== Elements determining job priority ==&lt;br /&gt;
In Mufasa, the priority of a job is computed by SLURM according to the following elements:&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#research users and students users|User category]]&amp;#039;&amp;#039;&amp;#039; (i.e., &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;)&lt;br /&gt;
::: Used to provide higher priority to jobs run by &amp;#039;&amp;#039;&amp;#039;research personnel&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#SLURM Quality of Service (QOS)|QOS]]&amp;#039;&amp;#039;&amp;#039; used by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;less resources&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Number of CPUs&amp;#039;&amp;#039;&amp;#039; requested by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;a lower number of CPUs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job Age&amp;#039;&amp;#039;&amp;#039;, i.e. the time that the job has been waiting in the queue&lt;br /&gt;
::: Used to provide higher priority to jobs which have been &amp;#039;&amp;#039;&amp;#039;queued for a longer time&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;FairShare&amp;#039;&amp;#039;&amp;#039;, i.e. a factor computed by SLURM to balance use of the system by different users&lt;br /&gt;
::: Used to provide higher priority to jobs by users who &amp;#039;&amp;#039;&amp;#039;used Mufasa&amp;#039;s resources less than others&amp;#039;&amp;#039;&amp;#039; (see [[#How FairShare works|below]])&lt;br /&gt;
&lt;br /&gt;
You can understand how the priority of your jobs depends on each of these factors using command&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sprio&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
A tailored version of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt; that only shows the parameters listed above is the following:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sprio -o &amp;quot;%6i %9u %.5Y %.5A %.5B %.5F %.5Q%.10T&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
Here is an example of its output:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
JOBID  USER      PRIOR   AGE ASSOC FAIRS   QOS      TRES&lt;br /&gt;
5774   itombini   3485   899  2500   136   750  cpu=-800&lt;br /&gt;
5780   fbottazzi  2259   413   625  1071   750  cpu=-600&lt;br /&gt;
5783   mzoccola   1759   210   625   974   750  cpu=-800&lt;br /&gt;
5790   amoglia    2710   149  2500   487   375  cpu=-800&lt;br /&gt;
5791   qliu       2225   131  2500    19   375  cpu=-800&lt;br /&gt;
5792   fpastori   2190    77  2500    39   375  cpu=-800&lt;br /&gt;
5793   bagostini  2027    11   625  1091  1500 cpu=-1200&lt;br /&gt;
5794   lpassoni   2510     2  2500    58   750  cpu=-800&lt;br /&gt;
5795   lpassoni   2510     2  2500    58   750  cpu=-800&lt;br /&gt;
5796   adurat     3094     2  2500   643   750  cpu=-800&lt;br /&gt;
In it, column ASSOC (i.e., association) corresponds to user category and column TRES corresponds to the number of CPUs. Column PRIOR provides the overall priority of each job.&lt;br /&gt;
&lt;br /&gt;
=== How FairShare works ===&lt;br /&gt;
In Mufasa 2.0, the FairShare of each user is computed according to these rules:&lt;br /&gt;
# It considers the quantity used by (current and past) user jobs of:&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;CPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of CPUs used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;RAM&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the amount of RAM used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;GPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of GPUs used&lt;br /&gt;
#** note: 40 GB GPUs have double the impact on FairShare than 20 GB GPUs&lt;br /&gt;
# The impact on FairShare of any use of resources is &amp;#039;&amp;#039;&amp;#039;proportional to its duration&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
#* example: using 32 GB of RAM for 48h has the same impact of using 64 GB of RAM for 24h&lt;br /&gt;
# FairShare has a &amp;quot;fading memory&amp;quot;: i.e., resource use has less and less impact on FairShare the farther in the past it is&lt;br /&gt;
#* consequence: over time, the &amp;quot;history&amp;quot; of a user gets forgotten by FairShare&lt;br /&gt;
&lt;br /&gt;
== How to maximise the priority of your jobs ==&lt;br /&gt;
&lt;br /&gt;
Every time you run a SLURM job, follow these guidelines:&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Choose the less powerful QOS compatible with the needs of your job&lt;br /&gt;
:: QOSes with access to less resources have higher base priority&lt;br /&gt;
&lt;br /&gt;
; Only request CPUs and RAM that your job really needs&lt;br /&gt;
:: Usually, code exploits multiple CPUs only if specifically designed to do so: if unsure, only ask for 1 CPU&lt;br /&gt;
:: The fewer resources your job asks for, the less it will wait before they become available&lt;br /&gt;
:: Asking for fewer CPUs and/or less RAM improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Do not request more time than your jobs needs to complete&lt;br /&gt;
:: Make a worst-case estimate and only ask for that duration&lt;br /&gt;
:: Asking for less of Mufasa&amp;#039;s time improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Debug and test code using less powerful QOSes before running it with more powerful QOSes&lt;br /&gt;
:: Your test jobs will get a higher priority and your FairShare will improve&lt;br /&gt;
&lt;br /&gt;
; Cancel jobs that you don&amp;#039;t need them anymore&lt;br /&gt;
:: Always use [[User_Jobs#Cancelling_a_job_with_scancel|scancel]] to delete completed (or crashed) jobs: your Fairshare will improve&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Protip: it&amp;#039;s a good idea to [[User Jobs#Looking for unused GPUs|check for unused GPUs]] before choosing what to request. Requesting a GPU that is currently idle will help your job get executed sooner.&lt;br /&gt;
&lt;br /&gt;
= SLURM partitions =&lt;br /&gt;
&lt;br /&gt;
Partitions are another mechanism provided by SLURM to create different levels of access to system resources. Since in Mufasa 2.0 access to resources is controlled via [[#SLURM Quality of Service (QOS)|QOSes]], partitions are not very relevant. (This is one of the main differences between Mufasa 1.0 and Mufasa 2.0.)&lt;br /&gt;
&lt;br /&gt;
Note, however, that the default values for some features of SLURM jobs (e.g., duration) are [[#Default values|set by the partition]].&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, there is a single SLURM partition, called &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt;, and all jobs run on it. The state of &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt; can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sinfo -o &amp;quot;%10P %5a %9T %11L %10l&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PARTITION  AVAIL STATE     DEFAULTTIME TIMELIMIT &lt;br /&gt;
jobs*      up    idle      1:00:00     3-00:00:00&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where columns correspond to the following information:&lt;br /&gt;
&lt;br /&gt;
:; PARTITION&lt;br /&gt;
:: name of the partition; the asterisks indicates that it&amp;#039;s the default one&lt;br /&gt;
&lt;br /&gt;
:; AVAIL&lt;br /&gt;
:: state/availability of the partition: see [[#Partition availability|below]]&lt;br /&gt;
&lt;br /&gt;
:; STATE&lt;br /&gt;
:: state (using [https://slurm.schedmd.com/sinfo.html#SECTION_NODE-STATE-CODES these codes])&lt;br /&gt;
:: typical values are &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;mixed&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that some of the resources are busy executing jobs while other are idle, and &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;allocated&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that all of the resources are in use&lt;br /&gt;
&lt;br /&gt;
:; DEFAULTTIME&lt;br /&gt;
:: default runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:; TIMELIMIT&lt;br /&gt;
:: maximum runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
The asterisk at the end of the partition name indicates the default partition, i.e. the one on which jobs which do not ask for a specific partition are run.&lt;br /&gt;
&lt;br /&gt;
Command &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; does not tell you about the &amp;#039;&amp;#039;jobs&amp;#039;&amp;#039; submitted to a partition. This information is obtained, instead, with [[User Jobs#Inspecting jobs with squeue|command &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;]].&lt;br /&gt;
&lt;br /&gt;
== Partition availability ==&lt;br /&gt;
&lt;br /&gt;
The most important information that &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; provides is the &amp;#039;&amp;#039;&amp;#039;availability&amp;#039;&amp;#039;&amp;#039; (also called &amp;#039;&amp;#039;&amp;#039;state&amp;#039;&amp;#039;&amp;#039;) of partitions. This is shown in column &amp;quot;AVAIL&amp;quot;. Possible partition states are:&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is available&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed as soon as resources allow&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is in the process of becoming unavailable (i.e., of entering the &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; state: see below)&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is unavailable&lt;br /&gt;
:: There are no running jobs&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
When a partition goes from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; no harm is done to running jobs. In a normally functioning SLURM system, the passage from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; happens only when no jobs are running on the partition. If (e.g., due to a malfunction) the passage happens with jobs still running, they get killed.&lt;br /&gt;
&lt;br /&gt;
A partition in state &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; requires intervention by a [[Roles|Job Administrator]] to be restored to &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
== Default values ==&lt;br /&gt;
&lt;br /&gt;
The features of SLURM partitions, including the &amp;#039;&amp;#039;&amp;#039;default values&amp;#039;&amp;#039;&amp;#039; which are applied to jobs that do not make explicit requests, can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scontrol show partition&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PartitionName=jobs&lt;br /&gt;
   AllowGroups=ALL AllowAccounts=ALL AllowQos=nogpu,gpulight,gpu,gpuwide,gpuheavy-20,gpuheavy-40&lt;br /&gt;
   AllocNodes=ALL Default=YES QoS=N/A&lt;br /&gt;
=&amp;gt; DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO&lt;br /&gt;
   MaxNodes=UNLIMITED MaxTime=3-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED&lt;br /&gt;
   Nodes=gn01&lt;br /&gt;
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO&lt;br /&gt;
   OverTimeLimit=NONE PreemptMode=OFF&lt;br /&gt;
   State=UP TotalCPUs=48 TotalNodes=1 SelectTypeParameters=NONE&lt;br /&gt;
   JobDefaults=(null)&lt;br /&gt;
=&amp;gt; DefMemPerNode=4096 MaxMemPerNode=UNLIMITED&lt;br /&gt;
   TRES=cpu=48,mem=1011435M,node=1,billing=49,gres/gpu=13,gres/gpu:3g.20gb=5,gres/gpu:40gb=3,gres/gpu:4g.20gb=5&lt;br /&gt;
   TRESBillingWeights=cpu=1.0,gres/gpu:3g.20gb=6.0,gres/gpu:4g.20gb=6.0,gres/gpu:40gb=6.0,mem=0.05g&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In the example, we have highlighted with  &amp;#039;&amp;#039;&amp;#039;=&amp;gt;&amp;#039;&amp;#039;&amp;#039;  the most relevant default values for Mufasa users, i.e.:&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefaultTime&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default execution time assigned to a job run on the partition (e.g., 1 hour)&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefMemPerNode&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default amount of RAM assigned to a job run on the partition (e.g., 4GB)&lt;br /&gt;
&lt;br /&gt;
= System resources subjected to limitations =&lt;br /&gt;
&lt;br /&gt;
In systems based on SLURM like Mufasa, &amp;#039;&amp;#039;&amp;#039;TRES (Trackable RESources)&amp;#039;&amp;#039;&amp;#039; are (from [https://slurm.schedmd.com/tres.html SLURM&amp;#039;s documentation] &amp;quot;&amp;#039;&amp;#039;resources that can be tracked for usage or used to enforce limits against.&amp;#039;&amp;#039;&amp;quot;&lt;br /&gt;
&lt;br /&gt;
TRES include CPUs, RAM and &amp;#039;&amp;#039;&amp;#039;GRES&amp;#039;&amp;#039;&amp;#039;. The last term stands for &amp;#039;&amp;#039;Generic RESources&amp;#039;&amp;#039; that a job may need for its execution. In Mufasa, the only &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; resources are the GPUs.&lt;br /&gt;
&lt;br /&gt;
== &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax ==&lt;br /&gt;
&lt;br /&gt;
To ask SLURM to assign GRES resources (i.e., GPUs) to a job, a special syntax must be used. Precisely, the name of each GPU resource takes the form&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:Name:Type&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
Considering the [[System#CPUs and GPUs|GPU complement of Mufasa]], &amp;lt;code&amp;gt;Type&amp;lt;/code&amp;gt; takes the following values:&lt;br /&gt;
&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:40gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 40 Gbytes of RAM&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:4g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 4 compute units&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 3 compute units&lt;br /&gt;
&lt;br /&gt;
So, for instance,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
identifies a resource corresponding to a GPU with 20 GB of RAM and 3 compute units.&lt;br /&gt;
&lt;br /&gt;
When asking for a GRES resource (e.g., in an &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command or an &amp;lt;code&amp;gt;SBATCH&amp;lt;/code&amp;gt; directive of an [[User Jobs#Using execution scripts to run jobs|execution script]]), the syntax required by SLURM is&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:&amp;lt;Type&amp;gt;:&amp;lt;Quantity&amp;gt;&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
where &amp;lt;code&amp;gt;Quantity&amp;lt;/code&amp;gt; is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of type &amp;lt;code&amp;gt;4g.20gb&amp;lt;/code&amp;gt; the syntax is&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:4g.20gb:2&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SLURM&amp;#039;s &amp;#039;&amp;#039;generic resources&amp;#039;&amp;#039; are defined in &amp;lt;code&amp;gt;/etc/slurm/gres.conf&amp;lt;/code&amp;gt;. In order to make GPUs available to SLURM&amp;#039;s &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; management, Mufasa makes use of Nvidia&amp;#039;s [https://developer.nvidia.com/nvidia-management-library-nvml NVML library]. For additional information see [https://slurm.schedmd.com/gres.html SLURM&amp;#039;s documentation].&lt;/div&gt;</summary>
		<author><name>GiulioFontana</name></author>
	</entry>
	<entry>
		<id>https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2562</id>
		<title>SLURM</title>
		<link rel="alternate" type="text/html" href="https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2562"/>
		<updated>2026-11-03T14:13:48Z</updated>

		<summary type="html">&lt;p&gt;GiulioFontana: /* Elements determining job priority */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page presents the features of SLURM that are most relevant to Mufasa&amp;#039;s [[Roles|Job Users]]. Job Users can submit jobs for execution, cancel their own jobs, and see other users&amp;#039; jobs (but not intervene on them).&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa &amp;#039;&amp;#039;&amp;#039;must use SLURM&amp;#039;&amp;#039;&amp;#039; to run resource-heavy processes, i.e. computing jobs that require one or more of the following:&lt;br /&gt;
* GPUs&lt;br /&gt;
* multiple CPUs&lt;br /&gt;
* powerful CPUs&lt;br /&gt;
* a significant amount of RAM&lt;br /&gt;
&lt;br /&gt;
In fact, only processes run via SLURM have access to all the resources of Mufasa. Processes run outside SLURM are executed by the [[System#Login server|login server]] virtual machine, which has minimal resources and no GPUs. Using SLURM is therefore the only way to execute resource-heavy jobs on Mufasa (this is a key difference between Mufasa 1.0 and Mufasa 2.0).&lt;br /&gt;
&lt;br /&gt;
= SLURM in a nutshell =&lt;br /&gt;
&lt;br /&gt;
Computation jobs on Mufasa needs to be launched via [[System#The SLURM job scheduling system|SLURM]]. SLURM provides jobs with access to the [[#System resources subjected to limitations|physical resources]] of Mufasa, such as CPUs, GPUs and RAM. Thanks to SLURM, processing jobs share system resources, optimising their occupation and availability. &lt;br /&gt;
&lt;br /&gt;
When a user runs a job, the job does not get executed immediately and is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039;. SLURM executes jobs according to their order in the queue: the top job in the queue gets executed as soon as the necessary resources are available, while jobs lower in the queue wait longer. The position of a job in the queue is due to the &amp;#039;&amp;#039;&amp;#039;[[#Job priority|priority]]&amp;#039;&amp;#039;&amp;#039; assigned to it by SLURM, with higher-priority jobs closer to the top. As a general rule:&lt;br /&gt;
&lt;br /&gt;
;: &amp;#039;&amp;#039;&amp;#039;the greater the fraction of Mufasa&amp;#039;s overall resources that a job asks for, the lower the job&amp;#039;s priority will be&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
The priority mechanism is used to encourage users to use Mufasa&amp;#039;s resources (i.e.: GPUs, CPUs, RAM, execution time) in an effective and equitable manner. This page includes a [[#How_to_maximise_the_priority_of_your_jobs|chart explaining how to maximise the priority of your jobs]].&lt;br /&gt;
&lt;br /&gt;
The &amp;#039;&amp;#039;&amp;#039;time&amp;#039;&amp;#039;&amp;#039; available to a job for its execution is controlled by SLURM. When a user requests execution of a job, they must specify the duration of the time slot that the job needs. The job must complete its execution before the end of the requested time slot, because at the end of such slot it gets killed by SLURM.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0 access to system resources is managed via SLURM&amp;#039;s [[#SLURM Quality of Service (QOS)|Quality of Service (QOS)]] mechanism (Mufasa 1.0 used [[#SLURM_partitions|partitions]] instead). While [[User Jobs#Running jobs with SLURM|launching a processing job via SLURM]], the user must always specify the chosen QOS. QOSes differ in the set of resources that they provide access to because each of them is designed to fit a given type of job.&lt;br /&gt;
&lt;br /&gt;
Mufasa sets limits to the number of jobs by the same user. This page includes a [[#Limits on jobs by the same user|table summarising such limits]].&lt;br /&gt;
&lt;br /&gt;
= SLURM Quality of Service (QOS) =&lt;br /&gt;
&lt;br /&gt;
Through &amp;#039;&amp;#039;&amp;#039;Quality of Services&amp;#039;&amp;#039;&amp;#039; (&amp;#039;&amp;#039;&amp;#039;QOSes&amp;#039;&amp;#039;&amp;#039;), SLURM lets system configurators assign a name to a set of related constraints.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, QOSes are used to define different levels of access to the server&amp;#039;s resources. When [[User Jobs|executing a job with SLURM]], a user must always &amp;#039;&amp;#039;&amp;#039;specify the QOS&amp;#039;&amp;#039;&amp;#039; that their job will use: this choice, in turn, determines what resources the job is able to access and influences the [[#Job priority|priority]] of the job.&lt;br /&gt;
&lt;br /&gt;
Mufasa&amp;#039;s QOSes and their features can be inspected with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list qos format=name%-11,priority,MaxSubmit,maxwall,maxtres%-80,mintres%-18&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following (see below for the same data in table form):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
Name          Priority MaxSubmit     MaxWall MaxTRES                                                                          MinTRES            &lt;br /&gt;
----------- ---------- --------- ----------- -------------------------------------------------------------------------------- ------------------ &lt;br /&gt;
normal               0                                                                                                                           &lt;br /&gt;
nogpu                4         1  3-00:00:00 cpu=16,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=128G                    &lt;br /&gt;
gpuheavy-20          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=2,mem=128G             gres/gpu:4g.20gb=1 &lt;br /&gt;
gpuheavy-40          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=1,gres/gpu:4g.20gb=0,mem=128G             gres/gpu:40gb=1    &lt;br /&gt;
gpulight             8         1    12:00:00 cpu=2,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpu                  2         1  1-00:00:00 cpu=8,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpuwide              2         2  1-00:00:00 cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=1,mem=64G              gres/gpu:4g.20gb=1 &lt;br /&gt;
build               32         1    02:00:00 cpu=2,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=16G&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The columns of this output are the following:&lt;br /&gt;
&lt;br /&gt;
:; Name&lt;br /&gt;
:: name of the QOS&lt;br /&gt;
&lt;br /&gt;
:; Priority&lt;br /&gt;
:: priority tier associated to the QOS (higher value = higher priority): see [[#Job priority|Job priority]] for details&lt;br /&gt;
&lt;br /&gt;
:; MaxSubmit&lt;br /&gt;
:: maximum number of jobs from a single user that can be submitted to SLURM with this QOS; submitted jobs include both running and queued jobs&lt;br /&gt;
:: See [[#Limits on jobs by the same user|Limits on jobs by the same user]] for an overview of the limits on jobs set by Mufasa.&lt;br /&gt;
&lt;br /&gt;
:; MaxWall&lt;br /&gt;
:: maximum wall clock duration of the jobs using the QOS (after which they are killed by SLURM), in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
:: For some QOSes these are not set: it means that they are determined by the [[#SLURM partitions|partition]]. Partitions also define the [[#Default values|default duration]] of jobs.&lt;br /&gt;
&lt;br /&gt;
:; MaxTRES&lt;br /&gt;
:: amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) available to a job using the QOS, where&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;cpu=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of CPUs (i.e., processor cores) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of CPUs specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;gres/&amp;#039;&amp;#039;gpu:Type&amp;#039;&amp;#039;=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of GPUs of class &amp;lt;code&amp;gt;&amp;#039;&amp;#039;Type&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; (see [[User Jobs#gres syntax|&amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax]]) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; (for QOSes that allow access to GPUs) if not specified, the job cannot be launched&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;mem=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;G&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum amount of system RAM is &amp;#039;&amp;#039;K&amp;#039;&amp;#039; GBytes&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of RAM specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
&lt;br /&gt;
:; MinTRES&lt;br /&gt;
:: minimum amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) that a job using the QOS must request in order to actually get executed by SLURM.&lt;br /&gt;
:: (If your job does not actually need these resources, you&amp;#039;ve chosen the wrong QOS: you can use one with a higher priority.)&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; QOS is the default one, and exists only to ensure that users always specify a QOS when running a job. Since &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; has zero priority and no resources, a job run using this QOS would never be run.&lt;br /&gt;
&lt;br /&gt;
The information provided by the &amp;lt;code&amp;gt;sacctmgr list qos&amp;lt;/code&amp;gt; command above is summarised by the following table:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! QOS&lt;br /&gt;
! Priority&amp;lt;/br&amp;gt;tier&lt;br /&gt;
! Max&amp;lt;/br&amp;gt;Submit&lt;br /&gt;
! MaxWall&amp;lt;/br&amp;gt;[h]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;CPUs&lt;br /&gt;
! max RAM&amp;lt;/br&amp;gt;[GB]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;3g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;4g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;40GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | build&lt;br /&gt;
| 32&lt;br /&gt;
| 1&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 16&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpulight&lt;br /&gt;
| 8&lt;br /&gt;
| 1&lt;br /&gt;
| 12&lt;br /&gt;
| 2&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | nogpu&lt;br /&gt;
| 4&lt;br /&gt;
| 1&lt;br /&gt;
| 72&lt;br /&gt;
| 16&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpu&lt;br /&gt;
| 2&lt;br /&gt;
| 1&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuwide&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-20&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| 2 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-40&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
A key piece of information in the table above is the &amp;#039;&amp;#039;&amp;#039;priority tier&amp;#039;&amp;#039;&amp;#039; associated to each QOS. In Mufasa 2.0, priority tiers are used to encourage users to use the &amp;#039;&amp;#039;&amp;#039;least powerful QOS that is compatible with their needs&amp;#039;&amp;#039;&amp;#039;, where &amp;quot;powerful&amp;quot; means &amp;quot;rich with resources&amp;quot;. Less powerful QOSes increase the priority of the jobs that use them, so these jobs tend to be executed sooner. See [[#Job priority|Job priority]] for details about how priority affects the execution order of jobs in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
== The &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS ==&lt;br /&gt;
&lt;br /&gt;
This QOS is specifically designed to be used by Mufasa users to &amp;#039;&amp;#039;&amp;#039;quickly build [[System#Containers|container images]]&amp;#039;&amp;#039;&amp;#039;. Its associated priority tier is very high: this, combined with the fact that jobs using this QOS require few resources, means that such jobs usually get executed very soon.&lt;br /&gt;
&lt;br /&gt;
On the other hand, the limited resources and the lack of access to the GPUs make the &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS unsuitable for other tasks.&lt;br /&gt;
&lt;br /&gt;
See [[Singularity#Building Singularity images|Building Singularity images]] for directions about building Singularity container images.&lt;br /&gt;
&lt;br /&gt;
== Restricted QOSes ==&lt;br /&gt;
&lt;br /&gt;
In Mufasa, the most powerful QOSes are reserved to researchers (called &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users: these include academic personnel and Ph.D. students). M.Sc. students, called &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users, cannot use them while running jobs.&lt;br /&gt;
&lt;br /&gt;
See [[#research users and students users|below]] to understand how &amp;#039;&amp;#039;user categories&amp;#039;&amp;#039; work in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
= &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users =&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa belong to two &amp;#039;&amp;#039;&amp;#039;user categories&amp;#039;&amp;#039;&amp;#039;. Categories provide users with different access to  to Mufasa&amp;#039;s resources: the idea being to provide researchers with more access while still letting students benefit from the server&amp;#039;s capabilities.&lt;br /&gt;
&lt;br /&gt;
User categories in Mufasa are the following:&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. academic personnel and Ph.D. students&lt;br /&gt;
::: * can use all QOSes, including the the [[#Restricted QOSes|restricted ones]]&lt;br /&gt;
::: * their jobs have a higher &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a higher number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. M.Sc. students&lt;br /&gt;
::: * cannot use the [[#Restricted QOSes|restricted QOSes]]&lt;br /&gt;
::: * their jobs have a lower &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a lower number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
You can inspect the differences in priority and running jobs between &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association format=&amp;quot;account,priority,maxjobs&amp;quot; | grep -E &amp;#039;Account|research|students&amp;#039;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
   Account   Priority MaxJobs &lt;br /&gt;
  research          4       2 &lt;br /&gt;
  students          1       1&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
To know what limits apply to your own user, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association where user=$USER format=&amp;quot;user,priority,maxjobs,qos%-60&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
      User   Priority MaxJobs QOS                                                          &lt;br /&gt;
---------- ---------- ------- ------------------------------------------------------------ &lt;br /&gt;
    preali          4       2 build,gpu,gpuheavy-20,gpuheavy-40,gpulight,gpuwide,nogpu&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The list under &amp;quot;QOS&amp;quot; shows what QOSes your user is allowed to use when [[User Jobs|running jobs]]. &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users can use all of them, while &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users can only access a subset of them.&lt;br /&gt;
&lt;br /&gt;
= Limits on jobs by the same user =&lt;br /&gt;
&lt;br /&gt;
Mufasa enforces limits on the number of jobs from a single user. Such limits aim at preventing users from &amp;quot;hogging&amp;quot; system resources, and apply to:&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;submitted jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that the user asked SLURM to execute, each of which may currently be either running or queued&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;running jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that are currently in execution&lt;br /&gt;
&lt;br /&gt;
Limits on submitted jobs are set via [[#SLURM Quality of Service (QOS)|QOSes]], while limits on running jobs are set via [[#research users and students users|user category]].&lt;br /&gt;
&lt;br /&gt;
The following table summarises what limits exist:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! &lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;running&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;submitted&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | global limits&amp;lt;br/&amp;gt;(system-wide)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to limits set by individual QOSes (below)&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | limits for&amp;lt;br/&amp;gt;individual&amp;lt;/br&amp;gt;QOSes&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to global limits (above)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for the&amp;#039;&amp;#039; &amp;lt;code&amp;gt;gpuwide&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;QOS&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for each of the other QOSes&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
= Job priority =&lt;br /&gt;
&lt;br /&gt;
Once the execution of a job has been requested, the job is not run immediately: it is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039; by SLURM, together with all the other jobs awaiting execution. The job on top of the queue at any time is the first to be put into execution as soon as the resources it requires are available. &lt;br /&gt;
&lt;br /&gt;
The order of the items in the job queue depends on their &amp;#039;&amp;#039;&amp;#039;priority&amp;#039;&amp;#039;&amp;#039;. The job with the highest priority is put by SLURM on top of the queue, while all other jobs are put in the queue in order of descending priority.&lt;br /&gt;
&lt;br /&gt;
The goal of SLURM is to maximise resource availability: i.e., to ensure the shorter possible wait time before job execution. To achieve this goal, in Mufasa SLURM is configured to &amp;#039;&amp;#039;&amp;#039;encourage users not to ask for resources or execution time that their job doesn&amp;#039;t need&amp;#039;&amp;#039;&amp;#039;. This is done via the priority mechanism: the more resources and/or time a job requests, the lower its priority will be; and the later it will be executed.&lt;br /&gt;
&lt;br /&gt;
Priority management in Mufasa is designed to set up a &amp;#039;&amp;#039;&amp;#039;virtuous cycle&amp;#039;&amp;#039;&amp;#039; where users, by carefully choosing what to ask for, obtain two results:&lt;br /&gt;
* they ensure that their job is executed as soon as possible;&lt;br /&gt;
* they leave as much as possible of Mufasa&amp;#039;s resources free for other users&amp;#039;s jobs.&lt;br /&gt;
&lt;br /&gt;
== Elements determining job priority ==&lt;br /&gt;
In Mufasa, the priority of a job is computed by SLURM according to the following elements:&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#research users and students users|User category]]&amp;#039;&amp;#039;&amp;#039; (i.e., &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;)&lt;br /&gt;
::: Used to provide higher priority to jobs run by &amp;#039;&amp;#039;&amp;#039;research personnel&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#SLURM Quality of Service (QOS)|QOS]]&amp;#039;&amp;#039;&amp;#039; used by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;less resources&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Number of CPUs&amp;#039;&amp;#039;&amp;#039; requested by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;a lower number of CPUs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job Age&amp;#039;&amp;#039;&amp;#039;, i.e. the time that the job has been waiting in the queue&lt;br /&gt;
::: Used to provide higher priority to jobs which have been &amp;#039;&amp;#039;&amp;#039;queued for a longer time&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;FairShare&amp;#039;&amp;#039;&amp;#039;, i.e. a factor computed by SLURM to balance use of the system by different users&lt;br /&gt;
::: Used to provide higher priority to jobs by users who &amp;#039;&amp;#039;&amp;#039;used Mufasa&amp;#039;s resources less than others&amp;#039;&amp;#039;&amp;#039; (see [[#How FairShare works|below]])&lt;br /&gt;
&lt;br /&gt;
You can understand how the priority of your jobs depends on each of these factors using command&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sprio&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
Here is an example of the output of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
          JOBID PARTITION   PRIORITY       SITE        AGE      ASSOC  FAIRSHARE    JOBSIZE        QOS&lt;br /&gt;
           5193 jobs            5677          0        905       2500         39       2155         78&lt;br /&gt;
           5202 jobs            3966          0        186        625        844       2155        156&lt;br /&gt;
           5203 jobs            5223          0        166       2500        325       2155         78&lt;br /&gt;
           5204 jobs            3744          0        158        625        649       2155        156&lt;br /&gt;
           5205 jobs            5186          0        141       2500        312       2155         78&lt;br /&gt;
           5232 jobs            4078          0         26        625        857       2414        156&lt;br /&gt;
           5236 jobs            4746          0          0       2500         13       2155         78&lt;br /&gt;
           5237 jobs            4834          0         10       2500         13       2155        156&lt;br /&gt;
           5238 jobs            5860          0         10       2500        312       2414        625&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
In it, column ASSOC (i.e., association) corresponds to user category. Column PRIORITY provides the overall priority of each job.&lt;br /&gt;
&lt;br /&gt;
=== How FairShare works ===&lt;br /&gt;
In Mufasa 2.0, the FairShare of each user is computed according to these rules:&lt;br /&gt;
# It considers the quantity used by (current and past) user jobs of:&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;CPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of CPUs used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;RAM&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the amount of RAM used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;GPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of GPUs used&lt;br /&gt;
#** note: 40 GB GPUs have double the impact on FairShare than 20 GB GPUs&lt;br /&gt;
# The impact on FairShare of any use of resources is &amp;#039;&amp;#039;&amp;#039;proportional to its duration&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
#* example: using 32 GB of RAM for 48h has the same impact of using 64 GB of RAM for 24h&lt;br /&gt;
# FairShare has a &amp;quot;fading memory&amp;quot;: i.e., resource use has less and less impact on FairShare the farther in the past it is&lt;br /&gt;
#* consequence: over time, the &amp;quot;history&amp;quot; of a user gets forgotten by FairShare&lt;br /&gt;
&lt;br /&gt;
== How to maximise the priority of your jobs ==&lt;br /&gt;
&lt;br /&gt;
Every time you run a SLURM job, follow these guidelines:&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Choose the less powerful QOS compatible with the needs of your job&lt;br /&gt;
:: QOSes with access to less resources have higher base priority&lt;br /&gt;
&lt;br /&gt;
; Only request CPUs and RAM that your job really needs&lt;br /&gt;
:: Usually, code exploits multiple CPUs only if specifically designed to do so: if unsure, only ask for 1 CPU&lt;br /&gt;
:: The fewer resources your job asks for, the less it will wait before they become available&lt;br /&gt;
:: Asking for fewer CPUs and/or less RAM improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Do not request more time than your jobs needs to complete&lt;br /&gt;
:: Make a worst-case estimate and only ask for that duration&lt;br /&gt;
:: Asking for less of Mufasa&amp;#039;s time improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Debug and test code using less powerful QOSes before running it with more powerful QOSes&lt;br /&gt;
:: Your test jobs will get a higher priority and your FairShare will improve&lt;br /&gt;
&lt;br /&gt;
; Cancel jobs that you don&amp;#039;t need them anymore&lt;br /&gt;
:: Always use [[User_Jobs#Cancelling_a_job_with_scancel|scancel]] to delete completed (or crashed) jobs: your Fairshare will improve&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Protip: it&amp;#039;s a good idea to [[User Jobs#Looking for unused GPUs|check for unused GPUs]] before choosing what to request. Requesting a GPU that is currently idle will help your job get executed sooner.&lt;br /&gt;
&lt;br /&gt;
= SLURM partitions =&lt;br /&gt;
&lt;br /&gt;
Partitions are another mechanism provided by SLURM to create different levels of access to system resources. Since in Mufasa 2.0 access to resources is controlled via [[#SLURM Quality of Service (QOS)|QOSes]], partitions are not very relevant. (This is one of the main differences between Mufasa 1.0 and Mufasa 2.0.)&lt;br /&gt;
&lt;br /&gt;
Note, however, that the default values for some features of SLURM jobs (e.g., duration) are [[#Default values|set by the partition]].&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, there is a single SLURM partition, called &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt;, and all jobs run on it. The state of &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt; can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sinfo -o &amp;quot;%10P %5a %9T %11L %10l&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PARTITION  AVAIL STATE     DEFAULTTIME TIMELIMIT &lt;br /&gt;
jobs*      up    idle      1:00:00     3-00:00:00&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where columns correspond to the following information:&lt;br /&gt;
&lt;br /&gt;
:; PARTITION&lt;br /&gt;
:: name of the partition; the asterisks indicates that it&amp;#039;s the default one&lt;br /&gt;
&lt;br /&gt;
:; AVAIL&lt;br /&gt;
:: state/availability of the partition: see [[#Partition availability|below]]&lt;br /&gt;
&lt;br /&gt;
:; STATE&lt;br /&gt;
:: state (using [https://slurm.schedmd.com/sinfo.html#SECTION_NODE-STATE-CODES these codes])&lt;br /&gt;
:: typical values are &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;mixed&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that some of the resources are busy executing jobs while other are idle, and &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;allocated&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that all of the resources are in use&lt;br /&gt;
&lt;br /&gt;
:; DEFAULTTIME&lt;br /&gt;
:: default runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:; TIMELIMIT&lt;br /&gt;
:: maximum runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
The asterisk at the end of the partition name indicates the default partition, i.e. the one on which jobs which do not ask for a specific partition are run.&lt;br /&gt;
&lt;br /&gt;
Command &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; does not tell you about the &amp;#039;&amp;#039;jobs&amp;#039;&amp;#039; submitted to a partition. This information is obtained, instead, with [[User Jobs#Inspecting jobs with squeue|command &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;]].&lt;br /&gt;
&lt;br /&gt;
== Partition availability ==&lt;br /&gt;
&lt;br /&gt;
The most important information that &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; provides is the &amp;#039;&amp;#039;&amp;#039;availability&amp;#039;&amp;#039;&amp;#039; (also called &amp;#039;&amp;#039;&amp;#039;state&amp;#039;&amp;#039;&amp;#039;) of partitions. This is shown in column &amp;quot;AVAIL&amp;quot;. Possible partition states are:&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is available&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed as soon as resources allow&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is in the process of becoming unavailable (i.e., of entering the &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; state: see below)&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is unavailable&lt;br /&gt;
:: There are no running jobs&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
When a partition goes from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; no harm is done to running jobs. In a normally functioning SLURM system, the passage from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; happens only when no jobs are running on the partition. If (e.g., due to a malfunction) the passage happens with jobs still running, they get killed.&lt;br /&gt;
&lt;br /&gt;
A partition in state &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; requires intervention by a [[Roles|Job Administrator]] to be restored to &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
== Default values ==&lt;br /&gt;
&lt;br /&gt;
The features of SLURM partitions, including the &amp;#039;&amp;#039;&amp;#039;default values&amp;#039;&amp;#039;&amp;#039; which are applied to jobs that do not make explicit requests, can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scontrol show partition&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PartitionName=jobs&lt;br /&gt;
   AllowGroups=ALL AllowAccounts=ALL AllowQos=nogpu,gpulight,gpu,gpuwide,gpuheavy-20,gpuheavy-40&lt;br /&gt;
   AllocNodes=ALL Default=YES QoS=N/A&lt;br /&gt;
=&amp;gt; DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO&lt;br /&gt;
   MaxNodes=UNLIMITED MaxTime=3-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED&lt;br /&gt;
   Nodes=gn01&lt;br /&gt;
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO&lt;br /&gt;
   OverTimeLimit=NONE PreemptMode=OFF&lt;br /&gt;
   State=UP TotalCPUs=48 TotalNodes=1 SelectTypeParameters=NONE&lt;br /&gt;
   JobDefaults=(null)&lt;br /&gt;
=&amp;gt; DefMemPerNode=4096 MaxMemPerNode=UNLIMITED&lt;br /&gt;
   TRES=cpu=48,mem=1011435M,node=1,billing=49,gres/gpu=13,gres/gpu:3g.20gb=5,gres/gpu:40gb=3,gres/gpu:4g.20gb=5&lt;br /&gt;
   TRESBillingWeights=cpu=1.0,gres/gpu:3g.20gb=6.0,gres/gpu:4g.20gb=6.0,gres/gpu:40gb=6.0,mem=0.05g&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In the example, we have highlighted with  &amp;#039;&amp;#039;&amp;#039;=&amp;gt;&amp;#039;&amp;#039;&amp;#039;  the most relevant default values for Mufasa users, i.e.:&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefaultTime&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default execution time assigned to a job run on the partition (e.g., 1 hour)&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefMemPerNode&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default amount of RAM assigned to a job run on the partition (e.g., 4GB)&lt;br /&gt;
&lt;br /&gt;
= System resources subjected to limitations =&lt;br /&gt;
&lt;br /&gt;
In systems based on SLURM like Mufasa, &amp;#039;&amp;#039;&amp;#039;TRES (Trackable RESources)&amp;#039;&amp;#039;&amp;#039; are (from [https://slurm.schedmd.com/tres.html SLURM&amp;#039;s documentation] &amp;quot;&amp;#039;&amp;#039;resources that can be tracked for usage or used to enforce limits against.&amp;#039;&amp;#039;&amp;quot;&lt;br /&gt;
&lt;br /&gt;
TRES include CPUs, RAM and &amp;#039;&amp;#039;&amp;#039;GRES&amp;#039;&amp;#039;&amp;#039;. The last term stands for &amp;#039;&amp;#039;Generic RESources&amp;#039;&amp;#039; that a job may need for its execution. In Mufasa, the only &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; resources are the GPUs.&lt;br /&gt;
&lt;br /&gt;
== &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax ==&lt;br /&gt;
&lt;br /&gt;
To ask SLURM to assign GRES resources (i.e., GPUs) to a job, a special syntax must be used. Precisely, the name of each GPU resource takes the form&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:Name:Type&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
Considering the [[System#CPUs and GPUs|GPU complement of Mufasa]], &amp;lt;code&amp;gt;Type&amp;lt;/code&amp;gt; takes the following values:&lt;br /&gt;
&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:40gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 40 Gbytes of RAM&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:4g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 4 compute units&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 3 compute units&lt;br /&gt;
&lt;br /&gt;
So, for instance,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
identifies a resource corresponding to a GPU with 20 GB of RAM and 3 compute units.&lt;br /&gt;
&lt;br /&gt;
When asking for a GRES resource (e.g., in an &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command or an &amp;lt;code&amp;gt;SBATCH&amp;lt;/code&amp;gt; directive of an [[User Jobs#Using execution scripts to run jobs|execution script]]), the syntax required by SLURM is&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:&amp;lt;Type&amp;gt;:&amp;lt;Quantity&amp;gt;&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
where &amp;lt;code&amp;gt;Quantity&amp;lt;/code&amp;gt; is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of type &amp;lt;code&amp;gt;4g.20gb&amp;lt;/code&amp;gt; the syntax is&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:4g.20gb:2&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SLURM&amp;#039;s &amp;#039;&amp;#039;generic resources&amp;#039;&amp;#039; are defined in &amp;lt;code&amp;gt;/etc/slurm/gres.conf&amp;lt;/code&amp;gt;. In order to make GPUs available to SLURM&amp;#039;s &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; management, Mufasa makes use of Nvidia&amp;#039;s [https://developer.nvidia.com/nvidia-management-library-nvml NVML library]. For additional information see [https://slurm.schedmd.com/gres.html SLURM&amp;#039;s documentation].&lt;/div&gt;</summary>
		<author><name>GiulioFontana</name></author>
	</entry>
	<entry>
		<id>https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2561</id>
		<title>SLURM</title>
		<link rel="alternate" type="text/html" href="https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2561"/>
		<updated>2026-10-20T15:01:06Z</updated>

		<summary type="html">&lt;p&gt;GiulioFontana: /* Elements determining job priority */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page presents the features of SLURM that are most relevant to Mufasa&amp;#039;s [[Roles|Job Users]]. Job Users can submit jobs for execution, cancel their own jobs, and see other users&amp;#039; jobs (but not intervene on them).&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa &amp;#039;&amp;#039;&amp;#039;must use SLURM&amp;#039;&amp;#039;&amp;#039; to run resource-heavy processes, i.e. computing jobs that require one or more of the following:&lt;br /&gt;
* GPUs&lt;br /&gt;
* multiple CPUs&lt;br /&gt;
* powerful CPUs&lt;br /&gt;
* a significant amount of RAM&lt;br /&gt;
&lt;br /&gt;
In fact, only processes run via SLURM have access to all the resources of Mufasa. Processes run outside SLURM are executed by the [[System#Login server|login server]] virtual machine, which has minimal resources and no GPUs. Using SLURM is therefore the only way to execute resource-heavy jobs on Mufasa (this is a key difference between Mufasa 1.0 and Mufasa 2.0).&lt;br /&gt;
&lt;br /&gt;
= SLURM in a nutshell =&lt;br /&gt;
&lt;br /&gt;
Computation jobs on Mufasa needs to be launched via [[System#The SLURM job scheduling system|SLURM]]. SLURM provides jobs with access to the [[#System resources subjected to limitations|physical resources]] of Mufasa, such as CPUs, GPUs and RAM. Thanks to SLURM, processing jobs share system resources, optimising their occupation and availability. &lt;br /&gt;
&lt;br /&gt;
When a user runs a job, the job does not get executed immediately and is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039;. SLURM executes jobs according to their order in the queue: the top job in the queue gets executed as soon as the necessary resources are available, while jobs lower in the queue wait longer. The position of a job in the queue is due to the &amp;#039;&amp;#039;&amp;#039;[[#Job priority|priority]]&amp;#039;&amp;#039;&amp;#039; assigned to it by SLURM, with higher-priority jobs closer to the top. As a general rule:&lt;br /&gt;
&lt;br /&gt;
;: &amp;#039;&amp;#039;&amp;#039;the greater the fraction of Mufasa&amp;#039;s overall resources that a job asks for, the lower the job&amp;#039;s priority will be&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
The priority mechanism is used to encourage users to use Mufasa&amp;#039;s resources (i.e.: GPUs, CPUs, RAM, execution time) in an effective and equitable manner. This page includes a [[#How_to_maximise_the_priority_of_your_jobs|chart explaining how to maximise the priority of your jobs]].&lt;br /&gt;
&lt;br /&gt;
The &amp;#039;&amp;#039;&amp;#039;time&amp;#039;&amp;#039;&amp;#039; available to a job for its execution is controlled by SLURM. When a user requests execution of a job, they must specify the duration of the time slot that the job needs. The job must complete its execution before the end of the requested time slot, because at the end of such slot it gets killed by SLURM.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0 access to system resources is managed via SLURM&amp;#039;s [[#SLURM Quality of Service (QOS)|Quality of Service (QOS)]] mechanism (Mufasa 1.0 used [[#SLURM_partitions|partitions]] instead). While [[User Jobs#Running jobs with SLURM|launching a processing job via SLURM]], the user must always specify the chosen QOS. QOSes differ in the set of resources that they provide access to because each of them is designed to fit a given type of job.&lt;br /&gt;
&lt;br /&gt;
Mufasa sets limits to the number of jobs by the same user. This page includes a [[#Limits on jobs by the same user|table summarising such limits]].&lt;br /&gt;
&lt;br /&gt;
= SLURM Quality of Service (QOS) =&lt;br /&gt;
&lt;br /&gt;
Through &amp;#039;&amp;#039;&amp;#039;Quality of Services&amp;#039;&amp;#039;&amp;#039; (&amp;#039;&amp;#039;&amp;#039;QOSes&amp;#039;&amp;#039;&amp;#039;), SLURM lets system configurators assign a name to a set of related constraints.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, QOSes are used to define different levels of access to the server&amp;#039;s resources. When [[User Jobs|executing a job with SLURM]], a user must always &amp;#039;&amp;#039;&amp;#039;specify the QOS&amp;#039;&amp;#039;&amp;#039; that their job will use: this choice, in turn, determines what resources the job is able to access and influences the [[#Job priority|priority]] of the job.&lt;br /&gt;
&lt;br /&gt;
Mufasa&amp;#039;s QOSes and their features can be inspected with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list qos format=name%-11,priority,MaxSubmit,maxwall,maxtres%-80,mintres%-18&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following (see below for the same data in table form):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
Name          Priority MaxSubmit     MaxWall MaxTRES                                                                          MinTRES            &lt;br /&gt;
----------- ---------- --------- ----------- -------------------------------------------------------------------------------- ------------------ &lt;br /&gt;
normal               0                                                                                                                           &lt;br /&gt;
nogpu                4         1  3-00:00:00 cpu=16,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=128G                    &lt;br /&gt;
gpuheavy-20          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=2,mem=128G             gres/gpu:4g.20gb=1 &lt;br /&gt;
gpuheavy-40          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=1,gres/gpu:4g.20gb=0,mem=128G             gres/gpu:40gb=1    &lt;br /&gt;
gpulight             8         1    12:00:00 cpu=2,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpu                  2         1  1-00:00:00 cpu=8,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpuwide              2         2  1-00:00:00 cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=1,mem=64G              gres/gpu:4g.20gb=1 &lt;br /&gt;
build               32         1    02:00:00 cpu=2,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=16G&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The columns of this output are the following:&lt;br /&gt;
&lt;br /&gt;
:; Name&lt;br /&gt;
:: name of the QOS&lt;br /&gt;
&lt;br /&gt;
:; Priority&lt;br /&gt;
:: priority tier associated to the QOS (higher value = higher priority): see [[#Job priority|Job priority]] for details&lt;br /&gt;
&lt;br /&gt;
:; MaxSubmit&lt;br /&gt;
:: maximum number of jobs from a single user that can be submitted to SLURM with this QOS; submitted jobs include both running and queued jobs&lt;br /&gt;
:: See [[#Limits on jobs by the same user|Limits on jobs by the same user]] for an overview of the limits on jobs set by Mufasa.&lt;br /&gt;
&lt;br /&gt;
:; MaxWall&lt;br /&gt;
:: maximum wall clock duration of the jobs using the QOS (after which they are killed by SLURM), in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
:: For some QOSes these are not set: it means that they are determined by the [[#SLURM partitions|partition]]. Partitions also define the [[#Default values|default duration]] of jobs.&lt;br /&gt;
&lt;br /&gt;
:; MaxTRES&lt;br /&gt;
:: amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) available to a job using the QOS, where&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;cpu=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of CPUs (i.e., processor cores) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of CPUs specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;gres/&amp;#039;&amp;#039;gpu:Type&amp;#039;&amp;#039;=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of GPUs of class &amp;lt;code&amp;gt;&amp;#039;&amp;#039;Type&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; (see [[User Jobs#gres syntax|&amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax]]) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; (for QOSes that allow access to GPUs) if not specified, the job cannot be launched&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;mem=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;G&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum amount of system RAM is &amp;#039;&amp;#039;K&amp;#039;&amp;#039; GBytes&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of RAM specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
&lt;br /&gt;
:; MinTRES&lt;br /&gt;
:: minimum amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) that a job using the QOS must request in order to actually get executed by SLURM.&lt;br /&gt;
:: (If your job does not actually need these resources, you&amp;#039;ve chosen the wrong QOS: you can use one with a higher priority.)&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; QOS is the default one, and exists only to ensure that users always specify a QOS when running a job. Since &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; has zero priority and no resources, a job run using this QOS would never be run.&lt;br /&gt;
&lt;br /&gt;
The information provided by the &amp;lt;code&amp;gt;sacctmgr list qos&amp;lt;/code&amp;gt; command above is summarised by the following table:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! QOS&lt;br /&gt;
! Priority&amp;lt;/br&amp;gt;tier&lt;br /&gt;
! Max&amp;lt;/br&amp;gt;Submit&lt;br /&gt;
! MaxWall&amp;lt;/br&amp;gt;[h]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;CPUs&lt;br /&gt;
! max RAM&amp;lt;/br&amp;gt;[GB]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;3g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;4g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;40GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | build&lt;br /&gt;
| 32&lt;br /&gt;
| 1&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 16&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpulight&lt;br /&gt;
| 8&lt;br /&gt;
| 1&lt;br /&gt;
| 12&lt;br /&gt;
| 2&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | nogpu&lt;br /&gt;
| 4&lt;br /&gt;
| 1&lt;br /&gt;
| 72&lt;br /&gt;
| 16&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpu&lt;br /&gt;
| 2&lt;br /&gt;
| 1&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuwide&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-20&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| 2 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-40&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
A key piece of information in the table above is the &amp;#039;&amp;#039;&amp;#039;priority tier&amp;#039;&amp;#039;&amp;#039; associated to each QOS. In Mufasa 2.0, priority tiers are used to encourage users to use the &amp;#039;&amp;#039;&amp;#039;least powerful QOS that is compatible with their needs&amp;#039;&amp;#039;&amp;#039;, where &amp;quot;powerful&amp;quot; means &amp;quot;rich with resources&amp;quot;. Less powerful QOSes increase the priority of the jobs that use them, so these jobs tend to be executed sooner. See [[#Job priority|Job priority]] for details about how priority affects the execution order of jobs in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
== The &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS ==&lt;br /&gt;
&lt;br /&gt;
This QOS is specifically designed to be used by Mufasa users to &amp;#039;&amp;#039;&amp;#039;quickly build [[System#Containers|container images]]&amp;#039;&amp;#039;&amp;#039;. Its associated priority tier is very high: this, combined with the fact that jobs using this QOS require few resources, means that such jobs usually get executed very soon.&lt;br /&gt;
&lt;br /&gt;
On the other hand, the limited resources and the lack of access to the GPUs make the &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS unsuitable for other tasks.&lt;br /&gt;
&lt;br /&gt;
See [[Singularity#Building Singularity images|Building Singularity images]] for directions about building Singularity container images.&lt;br /&gt;
&lt;br /&gt;
== Restricted QOSes ==&lt;br /&gt;
&lt;br /&gt;
In Mufasa, the most powerful QOSes are reserved to researchers (called &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users: these include academic personnel and Ph.D. students). M.Sc. students, called &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users, cannot use them while running jobs.&lt;br /&gt;
&lt;br /&gt;
See [[#research users and students users|below]] to understand how &amp;#039;&amp;#039;user categories&amp;#039;&amp;#039; work in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
= &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users =&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa belong to two &amp;#039;&amp;#039;&amp;#039;user categories&amp;#039;&amp;#039;&amp;#039;. Categories provide users with different access to  to Mufasa&amp;#039;s resources: the idea being to provide researchers with more access while still letting students benefit from the server&amp;#039;s capabilities.&lt;br /&gt;
&lt;br /&gt;
User categories in Mufasa are the following:&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. academic personnel and Ph.D. students&lt;br /&gt;
::: * can use all QOSes, including the the [[#Restricted QOSes|restricted ones]]&lt;br /&gt;
::: * their jobs have a higher &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a higher number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. M.Sc. students&lt;br /&gt;
::: * cannot use the [[#Restricted QOSes|restricted QOSes]]&lt;br /&gt;
::: * their jobs have a lower &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a lower number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
You can inspect the differences in priority and running jobs between &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association format=&amp;quot;account,priority,maxjobs&amp;quot; | grep -E &amp;#039;Account|research|students&amp;#039;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
   Account   Priority MaxJobs &lt;br /&gt;
  research          4       2 &lt;br /&gt;
  students          1       1&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
To know what limits apply to your own user, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association where user=$USER format=&amp;quot;user,priority,maxjobs,qos%-60&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
      User   Priority MaxJobs QOS                                                          &lt;br /&gt;
---------- ---------- ------- ------------------------------------------------------------ &lt;br /&gt;
    preali          4       2 build,gpu,gpuheavy-20,gpuheavy-40,gpulight,gpuwide,nogpu&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The list under &amp;quot;QOS&amp;quot; shows what QOSes your user is allowed to use when [[User Jobs|running jobs]]. &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users can use all of them, while &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users can only access a subset of them.&lt;br /&gt;
&lt;br /&gt;
= Limits on jobs by the same user =&lt;br /&gt;
&lt;br /&gt;
Mufasa enforces limits on the number of jobs from a single user. Such limits aim at preventing users from &amp;quot;hogging&amp;quot; system resources, and apply to:&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;submitted jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that the user asked SLURM to execute, each of which may currently be either running or queued&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;running jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that are currently in execution&lt;br /&gt;
&lt;br /&gt;
Limits on submitted jobs are set via [[#SLURM Quality of Service (QOS)|QOSes]], while limits on running jobs are set via [[#research users and students users|user category]].&lt;br /&gt;
&lt;br /&gt;
The following table summarises what limits exist:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! &lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;running&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;submitted&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | global limits&amp;lt;br/&amp;gt;(system-wide)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to limits set by individual QOSes (below)&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | limits for&amp;lt;br/&amp;gt;individual&amp;lt;/br&amp;gt;QOSes&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to global limits (above)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for the&amp;#039;&amp;#039; &amp;lt;code&amp;gt;gpuwide&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;QOS&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for each of the other QOSes&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
= Job priority =&lt;br /&gt;
&lt;br /&gt;
Once the execution of a job has been requested, the job is not run immediately: it is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039; by SLURM, together with all the other jobs awaiting execution. The job on top of the queue at any time is the first to be put into execution as soon as the resources it requires are available. &lt;br /&gt;
&lt;br /&gt;
The order of the items in the job queue depends on their &amp;#039;&amp;#039;&amp;#039;priority&amp;#039;&amp;#039;&amp;#039;. The job with the highest priority is put by SLURM on top of the queue, while all other jobs are put in the queue in order of descending priority.&lt;br /&gt;
&lt;br /&gt;
The goal of SLURM is to maximise resource availability: i.e., to ensure the shorter possible wait time before job execution. To achieve this goal, in Mufasa SLURM is configured to &amp;#039;&amp;#039;&amp;#039;encourage users not to ask for resources or execution time that their job doesn&amp;#039;t need&amp;#039;&amp;#039;&amp;#039;. This is done via the priority mechanism: the more resources and/or time a job requests, the lower its priority will be; and the later it will be executed.&lt;br /&gt;
&lt;br /&gt;
Priority management in Mufasa is designed to set up a &amp;#039;&amp;#039;&amp;#039;virtuous cycle&amp;#039;&amp;#039;&amp;#039; where users, by carefully choosing what to ask for, obtain two results:&lt;br /&gt;
* they ensure that their job is executed as soon as possible;&lt;br /&gt;
* they leave as much as possible of Mufasa&amp;#039;s resources free for other users&amp;#039;s jobs.&lt;br /&gt;
&lt;br /&gt;
== Elements determining job priority ==&lt;br /&gt;
In Mufasa, the priority of a job is computed by SLURM according to the following elements:&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#research users and students users|User category]]&amp;#039;&amp;#039;&amp;#039; (i.e., &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;)&lt;br /&gt;
::: Used to provide higher priority to jobs run by &amp;#039;&amp;#039;&amp;#039;research personnel&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#SLURM Quality of Service (QOS)|QOS]]&amp;#039;&amp;#039;&amp;#039; used by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;less resources&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Number of CPUs&amp;#039;&amp;#039;&amp;#039; requested by the job (also called &amp;quot;job size&amp;quot;)&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;a lower number of CPUs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job Age&amp;#039;&amp;#039;&amp;#039;, i.e. the time that the job has been waiting in the queue&lt;br /&gt;
::: Used to provide higher priority to jobs which have been &amp;#039;&amp;#039;&amp;#039;queued for a longer time&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;FairShare&amp;#039;&amp;#039;&amp;#039;, i.e. a factor computed by SLURM to balance use of the system by different users&lt;br /&gt;
::: Used to provide higher priority to jobs by users who &amp;#039;&amp;#039;&amp;#039;used Mufasa&amp;#039;s resources less than others&amp;#039;&amp;#039;&amp;#039; (see [[#How FairShare works|below]])&lt;br /&gt;
&lt;br /&gt;
You can understand how the priority of your jobs depends on each of these factors using command&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sprio&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
Here is an example of the output of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
          JOBID PARTITION   PRIORITY       SITE        AGE      ASSOC  FAIRSHARE    JOBSIZE        QOS&lt;br /&gt;
           5193 jobs            5677          0        905       2500         39       2155         78&lt;br /&gt;
           5202 jobs            3966          0        186        625        844       2155        156&lt;br /&gt;
           5203 jobs            5223          0        166       2500        325       2155         78&lt;br /&gt;
           5204 jobs            3744          0        158        625        649       2155        156&lt;br /&gt;
           5205 jobs            5186          0        141       2500        312       2155         78&lt;br /&gt;
           5232 jobs            4078          0         26        625        857       2414        156&lt;br /&gt;
           5236 jobs            4746          0          0       2500         13       2155         78&lt;br /&gt;
           5237 jobs            4834          0         10       2500         13       2155        156&lt;br /&gt;
           5238 jobs            5860          0         10       2500        312       2414        625&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
In it, column ASSOC (i.e., association) corresponds to user category. Column PRIORITY provides the overall priority of each job.&lt;br /&gt;
&lt;br /&gt;
=== How FairShare works ===&lt;br /&gt;
In Mufasa 2.0, the FairShare of each user is computed according to these rules:&lt;br /&gt;
# It considers the quantity used by (current and past) user jobs of:&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;CPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of CPUs used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;RAM&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the amount of RAM used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;GPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of GPUs used&lt;br /&gt;
#** note: 40 GB GPUs have double the impact on FairShare than 20 GB GPUs&lt;br /&gt;
# The impact on FairShare of any use of resources is &amp;#039;&amp;#039;&amp;#039;proportional to its duration&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
#* example: using 32 GB of RAM for 48h has the same impact of using 64 GB of RAM for 24h&lt;br /&gt;
# FairShare has a &amp;quot;fading memory&amp;quot;: i.e., resource use has less and less impact on FairShare the farther in the past it is&lt;br /&gt;
#* consequence: over time, the &amp;quot;history&amp;quot; of a user gets forgotten by FairShare&lt;br /&gt;
&lt;br /&gt;
== How to maximise the priority of your jobs ==&lt;br /&gt;
&lt;br /&gt;
Every time you run a SLURM job, follow these guidelines:&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Choose the less powerful QOS compatible with the needs of your job&lt;br /&gt;
:: QOSes with access to less resources have higher base priority&lt;br /&gt;
&lt;br /&gt;
; Only request CPUs and RAM that your job really needs&lt;br /&gt;
:: Usually, code exploits multiple CPUs only if specifically designed to do so: if unsure, only ask for 1 CPU&lt;br /&gt;
:: The fewer resources your job asks for, the less it will wait before they become available&lt;br /&gt;
:: Asking for fewer CPUs and/or less RAM improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Do not request more time than your jobs needs to complete&lt;br /&gt;
:: Make a worst-case estimate and only ask for that duration&lt;br /&gt;
:: Asking for less of Mufasa&amp;#039;s time improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Debug and test code using less powerful QOSes before running it with more powerful QOSes&lt;br /&gt;
:: Your test jobs will get a higher priority and your FairShare will improve&lt;br /&gt;
&lt;br /&gt;
; Cancel jobs that you don&amp;#039;t need them anymore&lt;br /&gt;
:: Always use [[User_Jobs#Cancelling_a_job_with_scancel|scancel]] to delete completed (or crashed) jobs: your Fairshare will improve&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Protip: it&amp;#039;s a good idea to [[User Jobs#Looking for unused GPUs|check for unused GPUs]] before choosing what to request. Requesting a GPU that is currently idle will help your job get executed sooner.&lt;br /&gt;
&lt;br /&gt;
= SLURM partitions =&lt;br /&gt;
&lt;br /&gt;
Partitions are another mechanism provided by SLURM to create different levels of access to system resources. Since in Mufasa 2.0 access to resources is controlled via [[#SLURM Quality of Service (QOS)|QOSes]], partitions are not very relevant. (This is one of the main differences between Mufasa 1.0 and Mufasa 2.0.)&lt;br /&gt;
&lt;br /&gt;
Note, however, that the default values for some features of SLURM jobs (e.g., duration) are [[#Default values|set by the partition]].&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, there is a single SLURM partition, called &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt;, and all jobs run on it. The state of &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt; can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sinfo -o &amp;quot;%10P %5a %9T %11L %10l&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PARTITION  AVAIL STATE     DEFAULTTIME TIMELIMIT &lt;br /&gt;
jobs*      up    idle      1:00:00     3-00:00:00&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where columns correspond to the following information:&lt;br /&gt;
&lt;br /&gt;
:; PARTITION&lt;br /&gt;
:: name of the partition; the asterisks indicates that it&amp;#039;s the default one&lt;br /&gt;
&lt;br /&gt;
:; AVAIL&lt;br /&gt;
:: state/availability of the partition: see [[#Partition availability|below]]&lt;br /&gt;
&lt;br /&gt;
:; STATE&lt;br /&gt;
:: state (using [https://slurm.schedmd.com/sinfo.html#SECTION_NODE-STATE-CODES these codes])&lt;br /&gt;
:: typical values are &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;mixed&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that some of the resources are busy executing jobs while other are idle, and &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;allocated&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that all of the resources are in use&lt;br /&gt;
&lt;br /&gt;
:; DEFAULTTIME&lt;br /&gt;
:: default runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:; TIMELIMIT&lt;br /&gt;
:: maximum runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
The asterisk at the end of the partition name indicates the default partition, i.e. the one on which jobs which do not ask for a specific partition are run.&lt;br /&gt;
&lt;br /&gt;
Command &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; does not tell you about the &amp;#039;&amp;#039;jobs&amp;#039;&amp;#039; submitted to a partition. This information is obtained, instead, with [[User Jobs#Inspecting jobs with squeue|command &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;]].&lt;br /&gt;
&lt;br /&gt;
== Partition availability ==&lt;br /&gt;
&lt;br /&gt;
The most important information that &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; provides is the &amp;#039;&amp;#039;&amp;#039;availability&amp;#039;&amp;#039;&amp;#039; (also called &amp;#039;&amp;#039;&amp;#039;state&amp;#039;&amp;#039;&amp;#039;) of partitions. This is shown in column &amp;quot;AVAIL&amp;quot;. Possible partition states are:&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is available&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed as soon as resources allow&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is in the process of becoming unavailable (i.e., of entering the &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; state: see below)&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is unavailable&lt;br /&gt;
:: There are no running jobs&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
When a partition goes from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; no harm is done to running jobs. In a normally functioning SLURM system, the passage from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; happens only when no jobs are running on the partition. If (e.g., due to a malfunction) the passage happens with jobs still running, they get killed.&lt;br /&gt;
&lt;br /&gt;
A partition in state &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; requires intervention by a [[Roles|Job Administrator]] to be restored to &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
== Default values ==&lt;br /&gt;
&lt;br /&gt;
The features of SLURM partitions, including the &amp;#039;&amp;#039;&amp;#039;default values&amp;#039;&amp;#039;&amp;#039; which are applied to jobs that do not make explicit requests, can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scontrol show partition&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PartitionName=jobs&lt;br /&gt;
   AllowGroups=ALL AllowAccounts=ALL AllowQos=nogpu,gpulight,gpu,gpuwide,gpuheavy-20,gpuheavy-40&lt;br /&gt;
   AllocNodes=ALL Default=YES QoS=N/A&lt;br /&gt;
=&amp;gt; DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO&lt;br /&gt;
   MaxNodes=UNLIMITED MaxTime=3-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED&lt;br /&gt;
   Nodes=gn01&lt;br /&gt;
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO&lt;br /&gt;
   OverTimeLimit=NONE PreemptMode=OFF&lt;br /&gt;
   State=UP TotalCPUs=48 TotalNodes=1 SelectTypeParameters=NONE&lt;br /&gt;
   JobDefaults=(null)&lt;br /&gt;
=&amp;gt; DefMemPerNode=4096 MaxMemPerNode=UNLIMITED&lt;br /&gt;
   TRES=cpu=48,mem=1011435M,node=1,billing=49,gres/gpu=13,gres/gpu:3g.20gb=5,gres/gpu:40gb=3,gres/gpu:4g.20gb=5&lt;br /&gt;
   TRESBillingWeights=cpu=1.0,gres/gpu:3g.20gb=6.0,gres/gpu:4g.20gb=6.0,gres/gpu:40gb=6.0,mem=0.05g&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In the example, we have highlighted with  &amp;#039;&amp;#039;&amp;#039;=&amp;gt;&amp;#039;&amp;#039;&amp;#039;  the most relevant default values for Mufasa users, i.e.:&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefaultTime&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default execution time assigned to a job run on the partition (e.g., 1 hour)&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefMemPerNode&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default amount of RAM assigned to a job run on the partition (e.g., 4GB)&lt;br /&gt;
&lt;br /&gt;
= System resources subjected to limitations =&lt;br /&gt;
&lt;br /&gt;
In systems based on SLURM like Mufasa, &amp;#039;&amp;#039;&amp;#039;TRES (Trackable RESources)&amp;#039;&amp;#039;&amp;#039; are (from [https://slurm.schedmd.com/tres.html SLURM&amp;#039;s documentation] &amp;quot;&amp;#039;&amp;#039;resources that can be tracked for usage or used to enforce limits against.&amp;#039;&amp;#039;&amp;quot;&lt;br /&gt;
&lt;br /&gt;
TRES include CPUs, RAM and &amp;#039;&amp;#039;&amp;#039;GRES&amp;#039;&amp;#039;&amp;#039;. The last term stands for &amp;#039;&amp;#039;Generic RESources&amp;#039;&amp;#039; that a job may need for its execution. In Mufasa, the only &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; resources are the GPUs.&lt;br /&gt;
&lt;br /&gt;
== &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax ==&lt;br /&gt;
&lt;br /&gt;
To ask SLURM to assign GRES resources (i.e., GPUs) to a job, a special syntax must be used. Precisely, the name of each GPU resource takes the form&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:Name:Type&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
Considering the [[System#CPUs and GPUs|GPU complement of Mufasa]], &amp;lt;code&amp;gt;Type&amp;lt;/code&amp;gt; takes the following values:&lt;br /&gt;
&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:40gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 40 Gbytes of RAM&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:4g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 4 compute units&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 3 compute units&lt;br /&gt;
&lt;br /&gt;
So, for instance,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
identifies a resource corresponding to a GPU with 20 GB of RAM and 3 compute units.&lt;br /&gt;
&lt;br /&gt;
When asking for a GRES resource (e.g., in an &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command or an &amp;lt;code&amp;gt;SBATCH&amp;lt;/code&amp;gt; directive of an [[User Jobs#Using execution scripts to run jobs|execution script]]), the syntax required by SLURM is&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:&amp;lt;Type&amp;gt;:&amp;lt;Quantity&amp;gt;&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
where &amp;lt;code&amp;gt;Quantity&amp;lt;/code&amp;gt; is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of type &amp;lt;code&amp;gt;4g.20gb&amp;lt;/code&amp;gt; the syntax is&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:4g.20gb:2&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SLURM&amp;#039;s &amp;#039;&amp;#039;generic resources&amp;#039;&amp;#039; are defined in &amp;lt;code&amp;gt;/etc/slurm/gres.conf&amp;lt;/code&amp;gt;. In order to make GPUs available to SLURM&amp;#039;s &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; management, Mufasa makes use of Nvidia&amp;#039;s [https://developer.nvidia.com/nvidia-management-library-nvml NVML library]. For additional information see [https://slurm.schedmd.com/gres.html SLURM&amp;#039;s documentation].&lt;/div&gt;</summary>
		<author><name>GiulioFontana</name></author>
	</entry>
	<entry>
		<id>https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2560</id>
		<title>SLURM</title>
		<link rel="alternate" type="text/html" href="https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2560"/>
		<updated>2026-10-20T14:25:54Z</updated>

		<summary type="html">&lt;p&gt;GiulioFontana: /* Elements determining job priority */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page presents the features of SLURM that are most relevant to Mufasa&amp;#039;s [[Roles|Job Users]]. Job Users can submit jobs for execution, cancel their own jobs, and see other users&amp;#039; jobs (but not intervene on them).&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa &amp;#039;&amp;#039;&amp;#039;must use SLURM&amp;#039;&amp;#039;&amp;#039; to run resource-heavy processes, i.e. computing jobs that require one or more of the following:&lt;br /&gt;
* GPUs&lt;br /&gt;
* multiple CPUs&lt;br /&gt;
* powerful CPUs&lt;br /&gt;
* a significant amount of RAM&lt;br /&gt;
&lt;br /&gt;
In fact, only processes run via SLURM have access to all the resources of Mufasa. Processes run outside SLURM are executed by the [[System#Login server|login server]] virtual machine, which has minimal resources and no GPUs. Using SLURM is therefore the only way to execute resource-heavy jobs on Mufasa (this is a key difference between Mufasa 1.0 and Mufasa 2.0).&lt;br /&gt;
&lt;br /&gt;
= SLURM in a nutshell =&lt;br /&gt;
&lt;br /&gt;
Computation jobs on Mufasa needs to be launched via [[System#The SLURM job scheduling system|SLURM]]. SLURM provides jobs with access to the [[#System resources subjected to limitations|physical resources]] of Mufasa, such as CPUs, GPUs and RAM. Thanks to SLURM, processing jobs share system resources, optimising their occupation and availability. &lt;br /&gt;
&lt;br /&gt;
When a user runs a job, the job does not get executed immediately and is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039;. SLURM executes jobs according to their order in the queue: the top job in the queue gets executed as soon as the necessary resources are available, while jobs lower in the queue wait longer. The position of a job in the queue is due to the &amp;#039;&amp;#039;&amp;#039;[[#Job priority|priority]]&amp;#039;&amp;#039;&amp;#039; assigned to it by SLURM, with higher-priority jobs closer to the top. As a general rule:&lt;br /&gt;
&lt;br /&gt;
;: &amp;#039;&amp;#039;&amp;#039;the greater the fraction of Mufasa&amp;#039;s overall resources that a job asks for, the lower the job&amp;#039;s priority will be&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
The priority mechanism is used to encourage users to use Mufasa&amp;#039;s resources (i.e.: GPUs, CPUs, RAM, execution time) in an effective and equitable manner. This page includes a [[#How_to_maximise_the_priority_of_your_jobs|chart explaining how to maximise the priority of your jobs]].&lt;br /&gt;
&lt;br /&gt;
The &amp;#039;&amp;#039;&amp;#039;time&amp;#039;&amp;#039;&amp;#039; available to a job for its execution is controlled by SLURM. When a user requests execution of a job, they must specify the duration of the time slot that the job needs. The job must complete its execution before the end of the requested time slot, because at the end of such slot it gets killed by SLURM.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0 access to system resources is managed via SLURM&amp;#039;s [[#SLURM Quality of Service (QOS)|Quality of Service (QOS)]] mechanism (Mufasa 1.0 used [[#SLURM_partitions|partitions]] instead). While [[User Jobs#Running jobs with SLURM|launching a processing job via SLURM]], the user must always specify the chosen QOS. QOSes differ in the set of resources that they provide access to because each of them is designed to fit a given type of job.&lt;br /&gt;
&lt;br /&gt;
Mufasa sets limits to the number of jobs by the same user. This page includes a [[#Limits on jobs by the same user|table summarising such limits]].&lt;br /&gt;
&lt;br /&gt;
= SLURM Quality of Service (QOS) =&lt;br /&gt;
&lt;br /&gt;
Through &amp;#039;&amp;#039;&amp;#039;Quality of Services&amp;#039;&amp;#039;&amp;#039; (&amp;#039;&amp;#039;&amp;#039;QOSes&amp;#039;&amp;#039;&amp;#039;), SLURM lets system configurators assign a name to a set of related constraints.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, QOSes are used to define different levels of access to the server&amp;#039;s resources. When [[User Jobs|executing a job with SLURM]], a user must always &amp;#039;&amp;#039;&amp;#039;specify the QOS&amp;#039;&amp;#039;&amp;#039; that their job will use: this choice, in turn, determines what resources the job is able to access and influences the [[#Job priority|priority]] of the job.&lt;br /&gt;
&lt;br /&gt;
Mufasa&amp;#039;s QOSes and their features can be inspected with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list qos format=name%-11,priority,MaxSubmit,maxwall,maxtres%-80,mintres%-18&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following (see below for the same data in table form):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
Name          Priority MaxSubmit     MaxWall MaxTRES                                                                          MinTRES            &lt;br /&gt;
----------- ---------- --------- ----------- -------------------------------------------------------------------------------- ------------------ &lt;br /&gt;
normal               0                                                                                                                           &lt;br /&gt;
nogpu                4         1  3-00:00:00 cpu=16,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=128G                    &lt;br /&gt;
gpuheavy-20          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=2,mem=128G             gres/gpu:4g.20gb=1 &lt;br /&gt;
gpuheavy-40          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=1,gres/gpu:4g.20gb=0,mem=128G             gres/gpu:40gb=1    &lt;br /&gt;
gpulight             8         1    12:00:00 cpu=2,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpu                  2         1  1-00:00:00 cpu=8,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpuwide              2         2  1-00:00:00 cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=1,mem=64G              gres/gpu:4g.20gb=1 &lt;br /&gt;
build               32         1    02:00:00 cpu=2,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=16G&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The columns of this output are the following:&lt;br /&gt;
&lt;br /&gt;
:; Name&lt;br /&gt;
:: name of the QOS&lt;br /&gt;
&lt;br /&gt;
:; Priority&lt;br /&gt;
:: priority tier associated to the QOS (higher value = higher priority): see [[#Job priority|Job priority]] for details&lt;br /&gt;
&lt;br /&gt;
:; MaxSubmit&lt;br /&gt;
:: maximum number of jobs from a single user that can be submitted to SLURM with this QOS; submitted jobs include both running and queued jobs&lt;br /&gt;
:: See [[#Limits on jobs by the same user|Limits on jobs by the same user]] for an overview of the limits on jobs set by Mufasa.&lt;br /&gt;
&lt;br /&gt;
:; MaxWall&lt;br /&gt;
:: maximum wall clock duration of the jobs using the QOS (after which they are killed by SLURM), in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
:: For some QOSes these are not set: it means that they are determined by the [[#SLURM partitions|partition]]. Partitions also define the [[#Default values|default duration]] of jobs.&lt;br /&gt;
&lt;br /&gt;
:; MaxTRES&lt;br /&gt;
:: amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) available to a job using the QOS, where&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;cpu=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of CPUs (i.e., processor cores) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of CPUs specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;gres/&amp;#039;&amp;#039;gpu:Type&amp;#039;&amp;#039;=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of GPUs of class &amp;lt;code&amp;gt;&amp;#039;&amp;#039;Type&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; (see [[User Jobs#gres syntax|&amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax]]) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; (for QOSes that allow access to GPUs) if not specified, the job cannot be launched&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;mem=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;G&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum amount of system RAM is &amp;#039;&amp;#039;K&amp;#039;&amp;#039; GBytes&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of RAM specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
&lt;br /&gt;
:; MinTRES&lt;br /&gt;
:: minimum amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) that a job using the QOS must request in order to actually get executed by SLURM.&lt;br /&gt;
:: (If your job does not actually need these resources, you&amp;#039;ve chosen the wrong QOS: you can use one with a higher priority.)&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; QOS is the default one, and exists only to ensure that users always specify a QOS when running a job. Since &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; has zero priority and no resources, a job run using this QOS would never be run.&lt;br /&gt;
&lt;br /&gt;
The information provided by the &amp;lt;code&amp;gt;sacctmgr list qos&amp;lt;/code&amp;gt; command above is summarised by the following table:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! QOS&lt;br /&gt;
! Priority&amp;lt;/br&amp;gt;tier&lt;br /&gt;
! Max&amp;lt;/br&amp;gt;Submit&lt;br /&gt;
! MaxWall&amp;lt;/br&amp;gt;[h]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;CPUs&lt;br /&gt;
! max RAM&amp;lt;/br&amp;gt;[GB]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;3g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;4g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;40GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | build&lt;br /&gt;
| 32&lt;br /&gt;
| 1&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 16&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpulight&lt;br /&gt;
| 8&lt;br /&gt;
| 1&lt;br /&gt;
| 12&lt;br /&gt;
| 2&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | nogpu&lt;br /&gt;
| 4&lt;br /&gt;
| 1&lt;br /&gt;
| 72&lt;br /&gt;
| 16&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpu&lt;br /&gt;
| 2&lt;br /&gt;
| 1&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuwide&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-20&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| 2 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-40&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
A key piece of information in the table above is the &amp;#039;&amp;#039;&amp;#039;priority tier&amp;#039;&amp;#039;&amp;#039; associated to each QOS. In Mufasa 2.0, priority tiers are used to encourage users to use the &amp;#039;&amp;#039;&amp;#039;least powerful QOS that is compatible with their needs&amp;#039;&amp;#039;&amp;#039;, where &amp;quot;powerful&amp;quot; means &amp;quot;rich with resources&amp;quot;. Less powerful QOSes increase the priority of the jobs that use them, so these jobs tend to be executed sooner. See [[#Job priority|Job priority]] for details about how priority affects the execution order of jobs in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
== The &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS ==&lt;br /&gt;
&lt;br /&gt;
This QOS is specifically designed to be used by Mufasa users to &amp;#039;&amp;#039;&amp;#039;quickly build [[System#Containers|container images]]&amp;#039;&amp;#039;&amp;#039;. Its associated priority tier is very high: this, combined with the fact that jobs using this QOS require few resources, means that such jobs usually get executed very soon.&lt;br /&gt;
&lt;br /&gt;
On the other hand, the limited resources and the lack of access to the GPUs make the &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS unsuitable for other tasks.&lt;br /&gt;
&lt;br /&gt;
See [[Singularity#Building Singularity images|Building Singularity images]] for directions about building Singularity container images.&lt;br /&gt;
&lt;br /&gt;
== Restricted QOSes ==&lt;br /&gt;
&lt;br /&gt;
In Mufasa, the most powerful QOSes are reserved to researchers (called &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users: these include academic personnel and Ph.D. students). M.Sc. students, called &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users, cannot use them while running jobs.&lt;br /&gt;
&lt;br /&gt;
See [[#research users and students users|below]] to understand how &amp;#039;&amp;#039;user categories&amp;#039;&amp;#039; work in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
= &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users =&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa belong to two &amp;#039;&amp;#039;&amp;#039;user categories&amp;#039;&amp;#039;&amp;#039;. Categories provide users with different access to  to Mufasa&amp;#039;s resources: the idea being to provide researchers with more access while still letting students benefit from the server&amp;#039;s capabilities.&lt;br /&gt;
&lt;br /&gt;
User categories in Mufasa are the following:&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. academic personnel and Ph.D. students&lt;br /&gt;
::: * can use all QOSes, including the the [[#Restricted QOSes|restricted ones]]&lt;br /&gt;
::: * their jobs have a higher &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a higher number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. M.Sc. students&lt;br /&gt;
::: * cannot use the [[#Restricted QOSes|restricted QOSes]]&lt;br /&gt;
::: * their jobs have a lower &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a lower number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
You can inspect the differences in priority and running jobs between &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association format=&amp;quot;account,priority,maxjobs&amp;quot; | grep -E &amp;#039;Account|research|students&amp;#039;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
   Account   Priority MaxJobs &lt;br /&gt;
  research          4       2 &lt;br /&gt;
  students          1       1&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
To know what limits apply to your own user, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association where user=$USER format=&amp;quot;user,priority,maxjobs,qos%-60&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
      User   Priority MaxJobs QOS                                                          &lt;br /&gt;
---------- ---------- ------- ------------------------------------------------------------ &lt;br /&gt;
    preali          4       2 build,gpu,gpuheavy-20,gpuheavy-40,gpulight,gpuwide,nogpu&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The list under &amp;quot;QOS&amp;quot; shows what QOSes your user is allowed to use when [[User Jobs|running jobs]]. &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users can use all of them, while &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users can only access a subset of them.&lt;br /&gt;
&lt;br /&gt;
= Limits on jobs by the same user =&lt;br /&gt;
&lt;br /&gt;
Mufasa enforces limits on the number of jobs from a single user. Such limits aim at preventing users from &amp;quot;hogging&amp;quot; system resources, and apply to:&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;submitted jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that the user asked SLURM to execute, each of which may currently be either running or queued&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;running jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that are currently in execution&lt;br /&gt;
&lt;br /&gt;
Limits on submitted jobs are set via [[#SLURM Quality of Service (QOS)|QOSes]], while limits on running jobs are set via [[#research users and students users|user category]].&lt;br /&gt;
&lt;br /&gt;
The following table summarises what limits exist:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! &lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;running&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;submitted&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | global limits&amp;lt;br/&amp;gt;(system-wide)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to limits set by individual QOSes (below)&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | limits for&amp;lt;br/&amp;gt;individual&amp;lt;/br&amp;gt;QOSes&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to global limits (above)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for the&amp;#039;&amp;#039; &amp;lt;code&amp;gt;gpuwide&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;QOS&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for each of the other QOSes&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
= Job priority =&lt;br /&gt;
&lt;br /&gt;
Once the execution of a job has been requested, the job is not run immediately: it is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039; by SLURM, together with all the other jobs awaiting execution. The job on top of the queue at any time is the first to be put into execution as soon as the resources it requires are available. &lt;br /&gt;
&lt;br /&gt;
The order of the items in the job queue depends on their &amp;#039;&amp;#039;&amp;#039;priority&amp;#039;&amp;#039;&amp;#039;. The job with the highest priority is put by SLURM on top of the queue, while all other jobs are put in the queue in order of descending priority.&lt;br /&gt;
&lt;br /&gt;
The goal of SLURM is to maximise resource availability: i.e., to ensure the shorter possible wait time before job execution. To achieve this goal, in Mufasa SLURM is configured to &amp;#039;&amp;#039;&amp;#039;encourage users not to ask for resources or execution time that their job doesn&amp;#039;t need&amp;#039;&amp;#039;&amp;#039;. This is done via the priority mechanism: the more resources and/or time a job requests, the lower its priority will be; and the later it will be executed.&lt;br /&gt;
&lt;br /&gt;
Priority management in Mufasa is designed to set up a &amp;#039;&amp;#039;&amp;#039;virtuous cycle&amp;#039;&amp;#039;&amp;#039; where users, by carefully choosing what to ask for, obtain two results:&lt;br /&gt;
* they ensure that their job is executed as soon as possible;&lt;br /&gt;
* they leave as much as possible of Mufasa&amp;#039;s resources free for other users&amp;#039;s jobs.&lt;br /&gt;
&lt;br /&gt;
== Elements determining job priority ==&lt;br /&gt;
In Mufasa, the priority of a job is computed by SLURM according to the following elements:&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#research users and students users|User category]]&amp;#039;&amp;#039;&amp;#039; (i.e., &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;)&lt;br /&gt;
::: Used to provide higher priority to jobs run by &amp;#039;&amp;#039;&amp;#039;research personnel&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#SLURM Quality of Service (QOS)|QOS]]&amp;#039;&amp;#039;&amp;#039; used by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;less resources&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Number of CPUs&amp;#039;&amp;#039;&amp;#039; requested by the job (also called &amp;quot;job size&amp;quot;)&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;a lower number of CPUs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job duration&amp;#039;&amp;#039;&amp;#039;, i.e. the execution time requested by the job&lt;br /&gt;
::: Used to provide higher priority to &amp;#039;&amp;#039;&amp;#039;shorter jobs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job Age&amp;#039;&amp;#039;&amp;#039;, i.e. the time that the job has been waiting in the queue&lt;br /&gt;
::: Used to provide higher priority to jobs which have been &amp;#039;&amp;#039;&amp;#039;queued for a longer time&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;FairShare&amp;#039;&amp;#039;&amp;#039;, i.e. a factor computed by SLURM to balance use of the system by different users&lt;br /&gt;
::: Used to provide higher priority to jobs by users who &amp;#039;&amp;#039;&amp;#039;used Mufasa&amp;#039;s resources less than others&amp;#039;&amp;#039;&amp;#039; (see [[#How FairShare works|below]])&lt;br /&gt;
&lt;br /&gt;
You can understand how the priority of your jobs depends on each of these factors using command&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sprio&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
Here is an example of the output of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
          JOBID PARTITION   PRIORITY       SITE        AGE      ASSOC  FAIRSHARE    JOBSIZE        QOS&lt;br /&gt;
           5193 jobs            5677          0        905       2500         39       2155         78&lt;br /&gt;
           5202 jobs            3966          0        186        625        844       2155        156&lt;br /&gt;
           5203 jobs            5223          0        166       2500        325       2155         78&lt;br /&gt;
           5204 jobs            3744          0        158        625        649       2155        156&lt;br /&gt;
           5205 jobs            5186          0        141       2500        312       2155         78&lt;br /&gt;
           5232 jobs            4078          0         26        625        857       2414        156&lt;br /&gt;
           5236 jobs            4746          0          0       2500         13       2155         78&lt;br /&gt;
           5237 jobs            4834          0         10       2500         13       2155        156&lt;br /&gt;
           5238 jobs            5860          0         10       2500        312       2414        625&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
In it, column ASSOC (i.e., association) corresponds to user category. Column PRIORITY provides the overall priority of each job.&lt;br /&gt;
&lt;br /&gt;
=== How FairShare works ===&lt;br /&gt;
In Mufasa 2.0, the FairShare of each user is computed according to these rules:&lt;br /&gt;
# It considers the quantity used by (current and past) user jobs of:&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;CPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of CPUs used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;RAM&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the amount of RAM used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;GPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of GPUs used&lt;br /&gt;
#** note: 40 GB GPUs have double the impact on FairShare than 20 GB GPUs&lt;br /&gt;
# The impact on FairShare of any use of resources is &amp;#039;&amp;#039;&amp;#039;proportional to its duration&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
#* example: using 32 GB of RAM for 48h has the same impact of using 64 GB of RAM for 24h&lt;br /&gt;
# FairShare has a &amp;quot;fading memory&amp;quot;: i.e., resource use has less and less impact on FairShare the farther in the past it is&lt;br /&gt;
#* consequence: over time, the &amp;quot;history&amp;quot; of a user gets forgotten by FairShare&lt;br /&gt;
&lt;br /&gt;
== How to maximise the priority of your jobs ==&lt;br /&gt;
&lt;br /&gt;
Every time you run a SLURM job, follow these guidelines:&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Choose the less powerful QOS compatible with the needs of your job&lt;br /&gt;
:: QOSes with access to less resources have higher base priority&lt;br /&gt;
&lt;br /&gt;
; Only request CPUs and RAM that your job really needs&lt;br /&gt;
:: Usually, code exploits multiple CPUs only if specifically designed to do so: if unsure, only ask for 1 CPU&lt;br /&gt;
:: The fewer resources your job asks for, the less it will wait before they become available&lt;br /&gt;
:: Asking for fewer CPUs and/or less RAM improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Do not request more time than your jobs needs to complete&lt;br /&gt;
:: Make a worst-case estimate and only ask for that duration&lt;br /&gt;
:: Asking for less of Mufasa&amp;#039;s time improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Debug and test code using less powerful QOSes before running it with more powerful QOSes&lt;br /&gt;
:: Your test jobs will get a higher priority and your FairShare will improve&lt;br /&gt;
&lt;br /&gt;
; Cancel jobs that you don&amp;#039;t need them anymore&lt;br /&gt;
:: Always use [[User_Jobs#Cancelling_a_job_with_scancel|scancel]] to delete completed (or crashed) jobs: your Fairshare will improve&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Protip: it&amp;#039;s a good idea to [[User Jobs#Looking for unused GPUs|check for unused GPUs]] before choosing what to request. Requesting a GPU that is currently idle will help your job get executed sooner.&lt;br /&gt;
&lt;br /&gt;
= SLURM partitions =&lt;br /&gt;
&lt;br /&gt;
Partitions are another mechanism provided by SLURM to create different levels of access to system resources. Since in Mufasa 2.0 access to resources is controlled via [[#SLURM Quality of Service (QOS)|QOSes]], partitions are not very relevant. (This is one of the main differences between Mufasa 1.0 and Mufasa 2.0.)&lt;br /&gt;
&lt;br /&gt;
Note, however, that the default values for some features of SLURM jobs (e.g., duration) are [[#Default values|set by the partition]].&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, there is a single SLURM partition, called &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt;, and all jobs run on it. The state of &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt; can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sinfo -o &amp;quot;%10P %5a %9T %11L %10l&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PARTITION  AVAIL STATE     DEFAULTTIME TIMELIMIT &lt;br /&gt;
jobs*      up    idle      1:00:00     3-00:00:00&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where columns correspond to the following information:&lt;br /&gt;
&lt;br /&gt;
:; PARTITION&lt;br /&gt;
:: name of the partition; the asterisks indicates that it&amp;#039;s the default one&lt;br /&gt;
&lt;br /&gt;
:; AVAIL&lt;br /&gt;
:: state/availability of the partition: see [[#Partition availability|below]]&lt;br /&gt;
&lt;br /&gt;
:; STATE&lt;br /&gt;
:: state (using [https://slurm.schedmd.com/sinfo.html#SECTION_NODE-STATE-CODES these codes])&lt;br /&gt;
:: typical values are &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;mixed&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that some of the resources are busy executing jobs while other are idle, and &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;allocated&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that all of the resources are in use&lt;br /&gt;
&lt;br /&gt;
:; DEFAULTTIME&lt;br /&gt;
:: default runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:; TIMELIMIT&lt;br /&gt;
:: maximum runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
The asterisk at the end of the partition name indicates the default partition, i.e. the one on which jobs which do not ask for a specific partition are run.&lt;br /&gt;
&lt;br /&gt;
Command &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; does not tell you about the &amp;#039;&amp;#039;jobs&amp;#039;&amp;#039; submitted to a partition. This information is obtained, instead, with [[User Jobs#Inspecting jobs with squeue|command &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;]].&lt;br /&gt;
&lt;br /&gt;
== Partition availability ==&lt;br /&gt;
&lt;br /&gt;
The most important information that &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; provides is the &amp;#039;&amp;#039;&amp;#039;availability&amp;#039;&amp;#039;&amp;#039; (also called &amp;#039;&amp;#039;&amp;#039;state&amp;#039;&amp;#039;&amp;#039;) of partitions. This is shown in column &amp;quot;AVAIL&amp;quot;. Possible partition states are:&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is available&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed as soon as resources allow&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is in the process of becoming unavailable (i.e., of entering the &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; state: see below)&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is unavailable&lt;br /&gt;
:: There are no running jobs&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
When a partition goes from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; no harm is done to running jobs. In a normally functioning SLURM system, the passage from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; happens only when no jobs are running on the partition. If (e.g., due to a malfunction) the passage happens with jobs still running, they get killed.&lt;br /&gt;
&lt;br /&gt;
A partition in state &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; requires intervention by a [[Roles|Job Administrator]] to be restored to &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
== Default values ==&lt;br /&gt;
&lt;br /&gt;
The features of SLURM partitions, including the &amp;#039;&amp;#039;&amp;#039;default values&amp;#039;&amp;#039;&amp;#039; which are applied to jobs that do not make explicit requests, can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scontrol show partition&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PartitionName=jobs&lt;br /&gt;
   AllowGroups=ALL AllowAccounts=ALL AllowQos=nogpu,gpulight,gpu,gpuwide,gpuheavy-20,gpuheavy-40&lt;br /&gt;
   AllocNodes=ALL Default=YES QoS=N/A&lt;br /&gt;
=&amp;gt; DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO&lt;br /&gt;
   MaxNodes=UNLIMITED MaxTime=3-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED&lt;br /&gt;
   Nodes=gn01&lt;br /&gt;
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO&lt;br /&gt;
   OverTimeLimit=NONE PreemptMode=OFF&lt;br /&gt;
   State=UP TotalCPUs=48 TotalNodes=1 SelectTypeParameters=NONE&lt;br /&gt;
   JobDefaults=(null)&lt;br /&gt;
=&amp;gt; DefMemPerNode=4096 MaxMemPerNode=UNLIMITED&lt;br /&gt;
   TRES=cpu=48,mem=1011435M,node=1,billing=49,gres/gpu=13,gres/gpu:3g.20gb=5,gres/gpu:40gb=3,gres/gpu:4g.20gb=5&lt;br /&gt;
   TRESBillingWeights=cpu=1.0,gres/gpu:3g.20gb=6.0,gres/gpu:4g.20gb=6.0,gres/gpu:40gb=6.0,mem=0.05g&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In the example, we have highlighted with  &amp;#039;&amp;#039;&amp;#039;=&amp;gt;&amp;#039;&amp;#039;&amp;#039;  the most relevant default values for Mufasa users, i.e.:&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefaultTime&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default execution time assigned to a job run on the partition (e.g., 1 hour)&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefMemPerNode&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default amount of RAM assigned to a job run on the partition (e.g., 4GB)&lt;br /&gt;
&lt;br /&gt;
= System resources subjected to limitations =&lt;br /&gt;
&lt;br /&gt;
In systems based on SLURM like Mufasa, &amp;#039;&amp;#039;&amp;#039;TRES (Trackable RESources)&amp;#039;&amp;#039;&amp;#039; are (from [https://slurm.schedmd.com/tres.html SLURM&amp;#039;s documentation] &amp;quot;&amp;#039;&amp;#039;resources that can be tracked for usage or used to enforce limits against.&amp;#039;&amp;#039;&amp;quot;&lt;br /&gt;
&lt;br /&gt;
TRES include CPUs, RAM and &amp;#039;&amp;#039;&amp;#039;GRES&amp;#039;&amp;#039;&amp;#039;. The last term stands for &amp;#039;&amp;#039;Generic RESources&amp;#039;&amp;#039; that a job may need for its execution. In Mufasa, the only &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; resources are the GPUs.&lt;br /&gt;
&lt;br /&gt;
== &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax ==&lt;br /&gt;
&lt;br /&gt;
To ask SLURM to assign GRES resources (i.e., GPUs) to a job, a special syntax must be used. Precisely, the name of each GPU resource takes the form&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:Name:Type&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
Considering the [[System#CPUs and GPUs|GPU complement of Mufasa]], &amp;lt;code&amp;gt;Type&amp;lt;/code&amp;gt; takes the following values:&lt;br /&gt;
&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:40gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 40 Gbytes of RAM&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:4g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 4 compute units&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 3 compute units&lt;br /&gt;
&lt;br /&gt;
So, for instance,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
identifies a resource corresponding to a GPU with 20 GB of RAM and 3 compute units.&lt;br /&gt;
&lt;br /&gt;
When asking for a GRES resource (e.g., in an &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command or an &amp;lt;code&amp;gt;SBATCH&amp;lt;/code&amp;gt; directive of an [[User Jobs#Using execution scripts to run jobs|execution script]]), the syntax required by SLURM is&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:&amp;lt;Type&amp;gt;:&amp;lt;Quantity&amp;gt;&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
where &amp;lt;code&amp;gt;Quantity&amp;lt;/code&amp;gt; is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of type &amp;lt;code&amp;gt;4g.20gb&amp;lt;/code&amp;gt; the syntax is&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:4g.20gb:2&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SLURM&amp;#039;s &amp;#039;&amp;#039;generic resources&amp;#039;&amp;#039; are defined in &amp;lt;code&amp;gt;/etc/slurm/gres.conf&amp;lt;/code&amp;gt;. In order to make GPUs available to SLURM&amp;#039;s &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; management, Mufasa makes use of Nvidia&amp;#039;s [https://developer.nvidia.com/nvidia-management-library-nvml NVML library]. For additional information see [https://slurm.schedmd.com/gres.html SLURM&amp;#039;s documentation].&lt;/div&gt;</summary>
		<author><name>GiulioFontana</name></author>
	</entry>
	<entry>
		<id>https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2559</id>
		<title>SLURM</title>
		<link rel="alternate" type="text/html" href="https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2559"/>
		<updated>2026-10-20T14:25:21Z</updated>

		<summary type="html">&lt;p&gt;GiulioFontana: /* Elements determining job priority */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page presents the features of SLURM that are most relevant to Mufasa&amp;#039;s [[Roles|Job Users]]. Job Users can submit jobs for execution, cancel their own jobs, and see other users&amp;#039; jobs (but not intervene on them).&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa &amp;#039;&amp;#039;&amp;#039;must use SLURM&amp;#039;&amp;#039;&amp;#039; to run resource-heavy processes, i.e. computing jobs that require one or more of the following:&lt;br /&gt;
* GPUs&lt;br /&gt;
* multiple CPUs&lt;br /&gt;
* powerful CPUs&lt;br /&gt;
* a significant amount of RAM&lt;br /&gt;
&lt;br /&gt;
In fact, only processes run via SLURM have access to all the resources of Mufasa. Processes run outside SLURM are executed by the [[System#Login server|login server]] virtual machine, which has minimal resources and no GPUs. Using SLURM is therefore the only way to execute resource-heavy jobs on Mufasa (this is a key difference between Mufasa 1.0 and Mufasa 2.0).&lt;br /&gt;
&lt;br /&gt;
= SLURM in a nutshell =&lt;br /&gt;
&lt;br /&gt;
Computation jobs on Mufasa needs to be launched via [[System#The SLURM job scheduling system|SLURM]]. SLURM provides jobs with access to the [[#System resources subjected to limitations|physical resources]] of Mufasa, such as CPUs, GPUs and RAM. Thanks to SLURM, processing jobs share system resources, optimising their occupation and availability. &lt;br /&gt;
&lt;br /&gt;
When a user runs a job, the job does not get executed immediately and is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039;. SLURM executes jobs according to their order in the queue: the top job in the queue gets executed as soon as the necessary resources are available, while jobs lower in the queue wait longer. The position of a job in the queue is due to the &amp;#039;&amp;#039;&amp;#039;[[#Job priority|priority]]&amp;#039;&amp;#039;&amp;#039; assigned to it by SLURM, with higher-priority jobs closer to the top. As a general rule:&lt;br /&gt;
&lt;br /&gt;
;: &amp;#039;&amp;#039;&amp;#039;the greater the fraction of Mufasa&amp;#039;s overall resources that a job asks for, the lower the job&amp;#039;s priority will be&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
The priority mechanism is used to encourage users to use Mufasa&amp;#039;s resources (i.e.: GPUs, CPUs, RAM, execution time) in an effective and equitable manner. This page includes a [[#How_to_maximise_the_priority_of_your_jobs|chart explaining how to maximise the priority of your jobs]].&lt;br /&gt;
&lt;br /&gt;
The &amp;#039;&amp;#039;&amp;#039;time&amp;#039;&amp;#039;&amp;#039; available to a job for its execution is controlled by SLURM. When a user requests execution of a job, they must specify the duration of the time slot that the job needs. The job must complete its execution before the end of the requested time slot, because at the end of such slot it gets killed by SLURM.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0 access to system resources is managed via SLURM&amp;#039;s [[#SLURM Quality of Service (QOS)|Quality of Service (QOS)]] mechanism (Mufasa 1.0 used [[#SLURM_partitions|partitions]] instead). While [[User Jobs#Running jobs with SLURM|launching a processing job via SLURM]], the user must always specify the chosen QOS. QOSes differ in the set of resources that they provide access to because each of them is designed to fit a given type of job.&lt;br /&gt;
&lt;br /&gt;
Mufasa sets limits to the number of jobs by the same user. This page includes a [[#Limits on jobs by the same user|table summarising such limits]].&lt;br /&gt;
&lt;br /&gt;
= SLURM Quality of Service (QOS) =&lt;br /&gt;
&lt;br /&gt;
Through &amp;#039;&amp;#039;&amp;#039;Quality of Services&amp;#039;&amp;#039;&amp;#039; (&amp;#039;&amp;#039;&amp;#039;QOSes&amp;#039;&amp;#039;&amp;#039;), SLURM lets system configurators assign a name to a set of related constraints.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, QOSes are used to define different levels of access to the server&amp;#039;s resources. When [[User Jobs|executing a job with SLURM]], a user must always &amp;#039;&amp;#039;&amp;#039;specify the QOS&amp;#039;&amp;#039;&amp;#039; that their job will use: this choice, in turn, determines what resources the job is able to access and influences the [[#Job priority|priority]] of the job.&lt;br /&gt;
&lt;br /&gt;
Mufasa&amp;#039;s QOSes and their features can be inspected with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list qos format=name%-11,priority,MaxSubmit,maxwall,maxtres%-80,mintres%-18&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following (see below for the same data in table form):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
Name          Priority MaxSubmit     MaxWall MaxTRES                                                                          MinTRES            &lt;br /&gt;
----------- ---------- --------- ----------- -------------------------------------------------------------------------------- ------------------ &lt;br /&gt;
normal               0                                                                                                                           &lt;br /&gt;
nogpu                4         1  3-00:00:00 cpu=16,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=128G                    &lt;br /&gt;
gpuheavy-20          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=2,mem=128G             gres/gpu:4g.20gb=1 &lt;br /&gt;
gpuheavy-40          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=1,gres/gpu:4g.20gb=0,mem=128G             gres/gpu:40gb=1    &lt;br /&gt;
gpulight             8         1    12:00:00 cpu=2,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpu                  2         1  1-00:00:00 cpu=8,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpuwide              2         2  1-00:00:00 cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=1,mem=64G              gres/gpu:4g.20gb=1 &lt;br /&gt;
build               32         1    02:00:00 cpu=2,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=16G&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The columns of this output are the following:&lt;br /&gt;
&lt;br /&gt;
:; Name&lt;br /&gt;
:: name of the QOS&lt;br /&gt;
&lt;br /&gt;
:; Priority&lt;br /&gt;
:: priority tier associated to the QOS (higher value = higher priority): see [[#Job priority|Job priority]] for details&lt;br /&gt;
&lt;br /&gt;
:; MaxSubmit&lt;br /&gt;
:: maximum number of jobs from a single user that can be submitted to SLURM with this QOS; submitted jobs include both running and queued jobs&lt;br /&gt;
:: See [[#Limits on jobs by the same user|Limits on jobs by the same user]] for an overview of the limits on jobs set by Mufasa.&lt;br /&gt;
&lt;br /&gt;
:; MaxWall&lt;br /&gt;
:: maximum wall clock duration of the jobs using the QOS (after which they are killed by SLURM), in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
:: For some QOSes these are not set: it means that they are determined by the [[#SLURM partitions|partition]]. Partitions also define the [[#Default values|default duration]] of jobs.&lt;br /&gt;
&lt;br /&gt;
:; MaxTRES&lt;br /&gt;
:: amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) available to a job using the QOS, where&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;cpu=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of CPUs (i.e., processor cores) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of CPUs specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;gres/&amp;#039;&amp;#039;gpu:Type&amp;#039;&amp;#039;=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of GPUs of class &amp;lt;code&amp;gt;&amp;#039;&amp;#039;Type&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; (see [[User Jobs#gres syntax|&amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax]]) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; (for QOSes that allow access to GPUs) if not specified, the job cannot be launched&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;mem=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;G&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum amount of system RAM is &amp;#039;&amp;#039;K&amp;#039;&amp;#039; GBytes&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of RAM specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
&lt;br /&gt;
:; MinTRES&lt;br /&gt;
:: minimum amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) that a job using the QOS must request in order to actually get executed by SLURM.&lt;br /&gt;
:: (If your job does not actually need these resources, you&amp;#039;ve chosen the wrong QOS: you can use one with a higher priority.)&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; QOS is the default one, and exists only to ensure that users always specify a QOS when running a job. Since &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; has zero priority and no resources, a job run using this QOS would never be run.&lt;br /&gt;
&lt;br /&gt;
The information provided by the &amp;lt;code&amp;gt;sacctmgr list qos&amp;lt;/code&amp;gt; command above is summarised by the following table:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! QOS&lt;br /&gt;
! Priority&amp;lt;/br&amp;gt;tier&lt;br /&gt;
! Max&amp;lt;/br&amp;gt;Submit&lt;br /&gt;
! MaxWall&amp;lt;/br&amp;gt;[h]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;CPUs&lt;br /&gt;
! max RAM&amp;lt;/br&amp;gt;[GB]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;3g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;4g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;40GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | build&lt;br /&gt;
| 32&lt;br /&gt;
| 1&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 16&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpulight&lt;br /&gt;
| 8&lt;br /&gt;
| 1&lt;br /&gt;
| 12&lt;br /&gt;
| 2&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | nogpu&lt;br /&gt;
| 4&lt;br /&gt;
| 1&lt;br /&gt;
| 72&lt;br /&gt;
| 16&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpu&lt;br /&gt;
| 2&lt;br /&gt;
| 1&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuwide&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-20&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| 2 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-40&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
A key piece of information in the table above is the &amp;#039;&amp;#039;&amp;#039;priority tier&amp;#039;&amp;#039;&amp;#039; associated to each QOS. In Mufasa 2.0, priority tiers are used to encourage users to use the &amp;#039;&amp;#039;&amp;#039;least powerful QOS that is compatible with their needs&amp;#039;&amp;#039;&amp;#039;, where &amp;quot;powerful&amp;quot; means &amp;quot;rich with resources&amp;quot;. Less powerful QOSes increase the priority of the jobs that use them, so these jobs tend to be executed sooner. See [[#Job priority|Job priority]] for details about how priority affects the execution order of jobs in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
== The &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS ==&lt;br /&gt;
&lt;br /&gt;
This QOS is specifically designed to be used by Mufasa users to &amp;#039;&amp;#039;&amp;#039;quickly build [[System#Containers|container images]]&amp;#039;&amp;#039;&amp;#039;. Its associated priority tier is very high: this, combined with the fact that jobs using this QOS require few resources, means that such jobs usually get executed very soon.&lt;br /&gt;
&lt;br /&gt;
On the other hand, the limited resources and the lack of access to the GPUs make the &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS unsuitable for other tasks.&lt;br /&gt;
&lt;br /&gt;
See [[Singularity#Building Singularity images|Building Singularity images]] for directions about building Singularity container images.&lt;br /&gt;
&lt;br /&gt;
== Restricted QOSes ==&lt;br /&gt;
&lt;br /&gt;
In Mufasa, the most powerful QOSes are reserved to researchers (called &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users: these include academic personnel and Ph.D. students). M.Sc. students, called &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users, cannot use them while running jobs.&lt;br /&gt;
&lt;br /&gt;
See [[#research users and students users|below]] to understand how &amp;#039;&amp;#039;user categories&amp;#039;&amp;#039; work in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
= &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users =&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa belong to two &amp;#039;&amp;#039;&amp;#039;user categories&amp;#039;&amp;#039;&amp;#039;. Categories provide users with different access to  to Mufasa&amp;#039;s resources: the idea being to provide researchers with more access while still letting students benefit from the server&amp;#039;s capabilities.&lt;br /&gt;
&lt;br /&gt;
User categories in Mufasa are the following:&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. academic personnel and Ph.D. students&lt;br /&gt;
::: * can use all QOSes, including the the [[#Restricted QOSes|restricted ones]]&lt;br /&gt;
::: * their jobs have a higher &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a higher number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. M.Sc. students&lt;br /&gt;
::: * cannot use the [[#Restricted QOSes|restricted QOSes]]&lt;br /&gt;
::: * their jobs have a lower &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a lower number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
You can inspect the differences in priority and running jobs between &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association format=&amp;quot;account,priority,maxjobs&amp;quot; | grep -E &amp;#039;Account|research|students&amp;#039;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
   Account   Priority MaxJobs &lt;br /&gt;
  research          4       2 &lt;br /&gt;
  students          1       1&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
To know what limits apply to your own user, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association where user=$USER format=&amp;quot;user,priority,maxjobs,qos%-60&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
      User   Priority MaxJobs QOS                                                          &lt;br /&gt;
---------- ---------- ------- ------------------------------------------------------------ &lt;br /&gt;
    preali          4       2 build,gpu,gpuheavy-20,gpuheavy-40,gpulight,gpuwide,nogpu&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The list under &amp;quot;QOS&amp;quot; shows what QOSes your user is allowed to use when [[User Jobs|running jobs]]. &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users can use all of them, while &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users can only access a subset of them.&lt;br /&gt;
&lt;br /&gt;
= Limits on jobs by the same user =&lt;br /&gt;
&lt;br /&gt;
Mufasa enforces limits on the number of jobs from a single user. Such limits aim at preventing users from &amp;quot;hogging&amp;quot; system resources, and apply to:&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;submitted jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that the user asked SLURM to execute, each of which may currently be either running or queued&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;running jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that are currently in execution&lt;br /&gt;
&lt;br /&gt;
Limits on submitted jobs are set via [[#SLURM Quality of Service (QOS)|QOSes]], while limits on running jobs are set via [[#research users and students users|user category]].&lt;br /&gt;
&lt;br /&gt;
The following table summarises what limits exist:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! &lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;running&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;submitted&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | global limits&amp;lt;br/&amp;gt;(system-wide)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to limits set by individual QOSes (below)&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | limits for&amp;lt;br/&amp;gt;individual&amp;lt;/br&amp;gt;QOSes&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to global limits (above)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for the&amp;#039;&amp;#039; &amp;lt;code&amp;gt;gpuwide&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;QOS&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for each of the other QOSes&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
= Job priority =&lt;br /&gt;
&lt;br /&gt;
Once the execution of a job has been requested, the job is not run immediately: it is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039; by SLURM, together with all the other jobs awaiting execution. The job on top of the queue at any time is the first to be put into execution as soon as the resources it requires are available. &lt;br /&gt;
&lt;br /&gt;
The order of the items in the job queue depends on their &amp;#039;&amp;#039;&amp;#039;priority&amp;#039;&amp;#039;&amp;#039;. The job with the highest priority is put by SLURM on top of the queue, while all other jobs are put in the queue in order of descending priority.&lt;br /&gt;
&lt;br /&gt;
The goal of SLURM is to maximise resource availability: i.e., to ensure the shorter possible wait time before job execution. To achieve this goal, in Mufasa SLURM is configured to &amp;#039;&amp;#039;&amp;#039;encourage users not to ask for resources or execution time that their job doesn&amp;#039;t need&amp;#039;&amp;#039;&amp;#039;. This is done via the priority mechanism: the more resources and/or time a job requests, the lower its priority will be; and the later it will be executed.&lt;br /&gt;
&lt;br /&gt;
Priority management in Mufasa is designed to set up a &amp;#039;&amp;#039;&amp;#039;virtuous cycle&amp;#039;&amp;#039;&amp;#039; where users, by carefully choosing what to ask for, obtain two results:&lt;br /&gt;
* they ensure that their job is executed as soon as possible;&lt;br /&gt;
* they leave as much as possible of Mufasa&amp;#039;s resources free for other users&amp;#039;s jobs.&lt;br /&gt;
&lt;br /&gt;
== Elements determining job priority ==&lt;br /&gt;
In Mufasa, the priority of a job is computed by SLURM according to the following elements:&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#research users and students users|User category]]&amp;#039;&amp;#039;&amp;#039; (i.e., &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;)&lt;br /&gt;
::: Used to provide higher priority to jobs run by &amp;#039;&amp;#039;&amp;#039;research personnel&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#SLURM Quality of Service (QOS)|QOS]]&amp;#039;&amp;#039;&amp;#039; used by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;less resources&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Number of CPUs&amp;#039;&amp;#039;&amp;#039; requested by the job (also called &amp;quot;job size&amp;quot;)&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;a lower number of CPUs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job duration&amp;#039;&amp;#039;&amp;#039;, i.e. the execution time requested by the job&lt;br /&gt;
::: Used to provide higher priority to &amp;#039;&amp;#039;&amp;#039;shorter jobs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job Age&amp;#039;&amp;#039;&amp;#039;, i.e. the time that the job has been waiting in the queue&lt;br /&gt;
::: Used to provide higher priority to jobs which have been &amp;#039;&amp;#039;&amp;#039;queued for a longer time&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;FairShare&amp;#039;&amp;#039;&amp;#039;, i.e. a factor computed by SLURM to balance use of the system by different users&lt;br /&gt;
::: Used to provide higher priority to jobs by users who &amp;#039;&amp;#039;&amp;#039;used Mufasa&amp;#039;s resources less than others&amp;#039;&amp;#039;&amp;#039; (see [[#How FairShare works|below]])&lt;br /&gt;
&lt;br /&gt;
You can understand how the priority of your jobs depends on each of these factors using command&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sprio&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
Here is an example of the output of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
          JOBID PARTITION   PRIORITY       SITE        AGE      ASSOC  FAIRSHARE    JOBSIZE        QOS&lt;br /&gt;
           5193 jobs            5987          0        871       2500         39       2499         78&lt;br /&gt;
           5201 jobs            4275          0        152        625        844       2498        156&lt;br /&gt;
           5202 jobs            4123          0          0        625        844       2498        156&lt;br /&gt;
           5203 jobs            5533          0        131       2500        325       2499         78&lt;br /&gt;
           5204 jobs            4052          0        124        625        649       2498        156&lt;br /&gt;
           5205 jobs            5494          0        107       2500        312       2498         78&lt;br /&gt;
           5212 jobs            4165          0         32        625        857       2495        156&lt;br /&gt;
           5231 jobs            5181          0          3       2500         26       2497        156&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
In it, column ASSOC (i.e., association) corresponds to user category. Column PRIORITY provides the overall priority of each job.&lt;br /&gt;
&lt;br /&gt;
=== How FairShare works ===&lt;br /&gt;
In Mufasa 2.0, the FairShare of each user is computed according to these rules:&lt;br /&gt;
# It considers the quantity used by (current and past) user jobs of:&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;CPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of CPUs used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;RAM&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the amount of RAM used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;GPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of GPUs used&lt;br /&gt;
#** note: 40 GB GPUs have double the impact on FairShare than 20 GB GPUs&lt;br /&gt;
# The impact on FairShare of any use of resources is &amp;#039;&amp;#039;&amp;#039;proportional to its duration&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
#* example: using 32 GB of RAM for 48h has the same impact of using 64 GB of RAM for 24h&lt;br /&gt;
# FairShare has a &amp;quot;fading memory&amp;quot;: i.e., resource use has less and less impact on FairShare the farther in the past it is&lt;br /&gt;
#* consequence: over time, the &amp;quot;history&amp;quot; of a user gets forgotten by FairShare&lt;br /&gt;
&lt;br /&gt;
== How to maximise the priority of your jobs ==&lt;br /&gt;
&lt;br /&gt;
Every time you run a SLURM job, follow these guidelines:&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Choose the less powerful QOS compatible with the needs of your job&lt;br /&gt;
:: QOSes with access to less resources have higher base priority&lt;br /&gt;
&lt;br /&gt;
; Only request CPUs and RAM that your job really needs&lt;br /&gt;
:: Usually, code exploits multiple CPUs only if specifically designed to do so: if unsure, only ask for 1 CPU&lt;br /&gt;
:: The fewer resources your job asks for, the less it will wait before they become available&lt;br /&gt;
:: Asking for fewer CPUs and/or less RAM improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Do not request more time than your jobs needs to complete&lt;br /&gt;
:: Make a worst-case estimate and only ask for that duration&lt;br /&gt;
:: Asking for less of Mufasa&amp;#039;s time improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Debug and test code using less powerful QOSes before running it with more powerful QOSes&lt;br /&gt;
:: Your test jobs will get a higher priority and your FairShare will improve&lt;br /&gt;
&lt;br /&gt;
; Cancel jobs that you don&amp;#039;t need them anymore&lt;br /&gt;
:: Always use [[User_Jobs#Cancelling_a_job_with_scancel|scancel]] to delete completed (or crashed) jobs: your Fairshare will improve&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Protip: it&amp;#039;s a good idea to [[User Jobs#Looking for unused GPUs|check for unused GPUs]] before choosing what to request. Requesting a GPU that is currently idle will help your job get executed sooner.&lt;br /&gt;
&lt;br /&gt;
= SLURM partitions =&lt;br /&gt;
&lt;br /&gt;
Partitions are another mechanism provided by SLURM to create different levels of access to system resources. Since in Mufasa 2.0 access to resources is controlled via [[#SLURM Quality of Service (QOS)|QOSes]], partitions are not very relevant. (This is one of the main differences between Mufasa 1.0 and Mufasa 2.0.)&lt;br /&gt;
&lt;br /&gt;
Note, however, that the default values for some features of SLURM jobs (e.g., duration) are [[#Default values|set by the partition]].&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, there is a single SLURM partition, called &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt;, and all jobs run on it. The state of &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt; can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sinfo -o &amp;quot;%10P %5a %9T %11L %10l&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PARTITION  AVAIL STATE     DEFAULTTIME TIMELIMIT &lt;br /&gt;
jobs*      up    idle      1:00:00     3-00:00:00&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where columns correspond to the following information:&lt;br /&gt;
&lt;br /&gt;
:; PARTITION&lt;br /&gt;
:: name of the partition; the asterisks indicates that it&amp;#039;s the default one&lt;br /&gt;
&lt;br /&gt;
:; AVAIL&lt;br /&gt;
:: state/availability of the partition: see [[#Partition availability|below]]&lt;br /&gt;
&lt;br /&gt;
:; STATE&lt;br /&gt;
:: state (using [https://slurm.schedmd.com/sinfo.html#SECTION_NODE-STATE-CODES these codes])&lt;br /&gt;
:: typical values are &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;mixed&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that some of the resources are busy executing jobs while other are idle, and &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;allocated&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that all of the resources are in use&lt;br /&gt;
&lt;br /&gt;
:; DEFAULTTIME&lt;br /&gt;
:: default runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:; TIMELIMIT&lt;br /&gt;
:: maximum runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
The asterisk at the end of the partition name indicates the default partition, i.e. the one on which jobs which do not ask for a specific partition are run.&lt;br /&gt;
&lt;br /&gt;
Command &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; does not tell you about the &amp;#039;&amp;#039;jobs&amp;#039;&amp;#039; submitted to a partition. This information is obtained, instead, with [[User Jobs#Inspecting jobs with squeue|command &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;]].&lt;br /&gt;
&lt;br /&gt;
== Partition availability ==&lt;br /&gt;
&lt;br /&gt;
The most important information that &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; provides is the &amp;#039;&amp;#039;&amp;#039;availability&amp;#039;&amp;#039;&amp;#039; (also called &amp;#039;&amp;#039;&amp;#039;state&amp;#039;&amp;#039;&amp;#039;) of partitions. This is shown in column &amp;quot;AVAIL&amp;quot;. Possible partition states are:&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is available&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed as soon as resources allow&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is in the process of becoming unavailable (i.e., of entering the &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; state: see below)&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is unavailable&lt;br /&gt;
:: There are no running jobs&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
When a partition goes from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; no harm is done to running jobs. In a normally functioning SLURM system, the passage from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; happens only when no jobs are running on the partition. If (e.g., due to a malfunction) the passage happens with jobs still running, they get killed.&lt;br /&gt;
&lt;br /&gt;
A partition in state &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; requires intervention by a [[Roles|Job Administrator]] to be restored to &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
== Default values ==&lt;br /&gt;
&lt;br /&gt;
The features of SLURM partitions, including the &amp;#039;&amp;#039;&amp;#039;default values&amp;#039;&amp;#039;&amp;#039; which are applied to jobs that do not make explicit requests, can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scontrol show partition&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PartitionName=jobs&lt;br /&gt;
   AllowGroups=ALL AllowAccounts=ALL AllowQos=nogpu,gpulight,gpu,gpuwide,gpuheavy-20,gpuheavy-40&lt;br /&gt;
   AllocNodes=ALL Default=YES QoS=N/A&lt;br /&gt;
=&amp;gt; DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO&lt;br /&gt;
   MaxNodes=UNLIMITED MaxTime=3-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED&lt;br /&gt;
   Nodes=gn01&lt;br /&gt;
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO&lt;br /&gt;
   OverTimeLimit=NONE PreemptMode=OFF&lt;br /&gt;
   State=UP TotalCPUs=48 TotalNodes=1 SelectTypeParameters=NONE&lt;br /&gt;
   JobDefaults=(null)&lt;br /&gt;
=&amp;gt; DefMemPerNode=4096 MaxMemPerNode=UNLIMITED&lt;br /&gt;
   TRES=cpu=48,mem=1011435M,node=1,billing=49,gres/gpu=13,gres/gpu:3g.20gb=5,gres/gpu:40gb=3,gres/gpu:4g.20gb=5&lt;br /&gt;
   TRESBillingWeights=cpu=1.0,gres/gpu:3g.20gb=6.0,gres/gpu:4g.20gb=6.0,gres/gpu:40gb=6.0,mem=0.05g&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In the example, we have highlighted with  &amp;#039;&amp;#039;&amp;#039;=&amp;gt;&amp;#039;&amp;#039;&amp;#039;  the most relevant default values for Mufasa users, i.e.:&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefaultTime&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default execution time assigned to a job run on the partition (e.g., 1 hour)&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefMemPerNode&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default amount of RAM assigned to a job run on the partition (e.g., 4GB)&lt;br /&gt;
&lt;br /&gt;
= System resources subjected to limitations =&lt;br /&gt;
&lt;br /&gt;
In systems based on SLURM like Mufasa, &amp;#039;&amp;#039;&amp;#039;TRES (Trackable RESources)&amp;#039;&amp;#039;&amp;#039; are (from [https://slurm.schedmd.com/tres.html SLURM&amp;#039;s documentation] &amp;quot;&amp;#039;&amp;#039;resources that can be tracked for usage or used to enforce limits against.&amp;#039;&amp;#039;&amp;quot;&lt;br /&gt;
&lt;br /&gt;
TRES include CPUs, RAM and &amp;#039;&amp;#039;&amp;#039;GRES&amp;#039;&amp;#039;&amp;#039;. The last term stands for &amp;#039;&amp;#039;Generic RESources&amp;#039;&amp;#039; that a job may need for its execution. In Mufasa, the only &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; resources are the GPUs.&lt;br /&gt;
&lt;br /&gt;
== &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax ==&lt;br /&gt;
&lt;br /&gt;
To ask SLURM to assign GRES resources (i.e., GPUs) to a job, a special syntax must be used. Precisely, the name of each GPU resource takes the form&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:Name:Type&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
Considering the [[System#CPUs and GPUs|GPU complement of Mufasa]], &amp;lt;code&amp;gt;Type&amp;lt;/code&amp;gt; takes the following values:&lt;br /&gt;
&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:40gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 40 Gbytes of RAM&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:4g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 4 compute units&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 3 compute units&lt;br /&gt;
&lt;br /&gt;
So, for instance,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
identifies a resource corresponding to a GPU with 20 GB of RAM and 3 compute units.&lt;br /&gt;
&lt;br /&gt;
When asking for a GRES resource (e.g., in an &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command or an &amp;lt;code&amp;gt;SBATCH&amp;lt;/code&amp;gt; directive of an [[User Jobs#Using execution scripts to run jobs|execution script]]), the syntax required by SLURM is&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:&amp;lt;Type&amp;gt;:&amp;lt;Quantity&amp;gt;&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
where &amp;lt;code&amp;gt;Quantity&amp;lt;/code&amp;gt; is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of type &amp;lt;code&amp;gt;4g.20gb&amp;lt;/code&amp;gt; the syntax is&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:4g.20gb:2&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SLURM&amp;#039;s &amp;#039;&amp;#039;generic resources&amp;#039;&amp;#039; are defined in &amp;lt;code&amp;gt;/etc/slurm/gres.conf&amp;lt;/code&amp;gt;. In order to make GPUs available to SLURM&amp;#039;s &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; management, Mufasa makes use of Nvidia&amp;#039;s [https://developer.nvidia.com/nvidia-management-library-nvml NVML library]. For additional information see [https://slurm.schedmd.com/gres.html SLURM&amp;#039;s documentation].&lt;/div&gt;</summary>
		<author><name>GiulioFontana</name></author>
	</entry>
	<entry>
		<id>https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2558</id>
		<title>SLURM</title>
		<link rel="alternate" type="text/html" href="https://biohpc.deib.polimi.it/index.php?title=SLURM&amp;diff=2558"/>
		<updated>2026-10-20T13:24:51Z</updated>

		<summary type="html">&lt;p&gt;GiulioFontana: /* Elements determining job priority */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page presents the features of SLURM that are most relevant to Mufasa&amp;#039;s [[Roles|Job Users]]. Job Users can submit jobs for execution, cancel their own jobs, and see other users&amp;#039; jobs (but not intervene on them).&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa &amp;#039;&amp;#039;&amp;#039;must use SLURM&amp;#039;&amp;#039;&amp;#039; to run resource-heavy processes, i.e. computing jobs that require one or more of the following:&lt;br /&gt;
* GPUs&lt;br /&gt;
* multiple CPUs&lt;br /&gt;
* powerful CPUs&lt;br /&gt;
* a significant amount of RAM&lt;br /&gt;
&lt;br /&gt;
In fact, only processes run via SLURM have access to all the resources of Mufasa. Processes run outside SLURM are executed by the [[System#Login server|login server]] virtual machine, which has minimal resources and no GPUs. Using SLURM is therefore the only way to execute resource-heavy jobs on Mufasa (this is a key difference between Mufasa 1.0 and Mufasa 2.0).&lt;br /&gt;
&lt;br /&gt;
= SLURM in a nutshell =&lt;br /&gt;
&lt;br /&gt;
Computation jobs on Mufasa needs to be launched via [[System#The SLURM job scheduling system|SLURM]]. SLURM provides jobs with access to the [[#System resources subjected to limitations|physical resources]] of Mufasa, such as CPUs, GPUs and RAM. Thanks to SLURM, processing jobs share system resources, optimising their occupation and availability. &lt;br /&gt;
&lt;br /&gt;
When a user runs a job, the job does not get executed immediately and is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039;. SLURM executes jobs according to their order in the queue: the top job in the queue gets executed as soon as the necessary resources are available, while jobs lower in the queue wait longer. The position of a job in the queue is due to the &amp;#039;&amp;#039;&amp;#039;[[#Job priority|priority]]&amp;#039;&amp;#039;&amp;#039; assigned to it by SLURM, with higher-priority jobs closer to the top. As a general rule:&lt;br /&gt;
&lt;br /&gt;
;: &amp;#039;&amp;#039;&amp;#039;the greater the fraction of Mufasa&amp;#039;s overall resources that a job asks for, the lower the job&amp;#039;s priority will be&amp;#039;&amp;#039;&amp;#039;.&lt;br /&gt;
&lt;br /&gt;
The priority mechanism is used to encourage users to use Mufasa&amp;#039;s resources (i.e.: GPUs, CPUs, RAM, execution time) in an effective and equitable manner. This page includes a [[#How_to_maximise_the_priority_of_your_jobs|chart explaining how to maximise the priority of your jobs]].&lt;br /&gt;
&lt;br /&gt;
The &amp;#039;&amp;#039;&amp;#039;time&amp;#039;&amp;#039;&amp;#039; available to a job for its execution is controlled by SLURM. When a user requests execution of a job, they must specify the duration of the time slot that the job needs. The job must complete its execution before the end of the requested time slot, because at the end of such slot it gets killed by SLURM.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0 access to system resources is managed via SLURM&amp;#039;s [[#SLURM Quality of Service (QOS)|Quality of Service (QOS)]] mechanism (Mufasa 1.0 used [[#SLURM_partitions|partitions]] instead). While [[User Jobs#Running jobs with SLURM|launching a processing job via SLURM]], the user must always specify the chosen QOS. QOSes differ in the set of resources that they provide access to because each of them is designed to fit a given type of job.&lt;br /&gt;
&lt;br /&gt;
Mufasa sets limits to the number of jobs by the same user. This page includes a [[#Limits on jobs by the same user|table summarising such limits]].&lt;br /&gt;
&lt;br /&gt;
= SLURM Quality of Service (QOS) =&lt;br /&gt;
&lt;br /&gt;
Through &amp;#039;&amp;#039;&amp;#039;Quality of Services&amp;#039;&amp;#039;&amp;#039; (&amp;#039;&amp;#039;&amp;#039;QOSes&amp;#039;&amp;#039;&amp;#039;), SLURM lets system configurators assign a name to a set of related constraints.&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, QOSes are used to define different levels of access to the server&amp;#039;s resources. When [[User Jobs|executing a job with SLURM]], a user must always &amp;#039;&amp;#039;&amp;#039;specify the QOS&amp;#039;&amp;#039;&amp;#039; that their job will use: this choice, in turn, determines what resources the job is able to access and influences the [[#Job priority|priority]] of the job.&lt;br /&gt;
&lt;br /&gt;
Mufasa&amp;#039;s QOSes and their features can be inspected with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list qos format=name%-11,priority,MaxSubmit,maxwall,maxtres%-80,mintres%-18&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following (see below for the same data in table form):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
Name          Priority MaxSubmit     MaxWall MaxTRES                                                                          MinTRES            &lt;br /&gt;
----------- ---------- --------- ----------- -------------------------------------------------------------------------------- ------------------ &lt;br /&gt;
normal               0                                                                                                                           &lt;br /&gt;
nogpu                4         1  3-00:00:00 cpu=16,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=128G                    &lt;br /&gt;
gpuheavy-20          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=2,mem=128G             gres/gpu:4g.20gb=1 &lt;br /&gt;
gpuheavy-40          1         1             cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=1,gres/gpu:4g.20gb=0,mem=128G             gres/gpu:40gb=1    &lt;br /&gt;
gpulight             8         1    12:00:00 cpu=2,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpu                  2         1  1-00:00:00 cpu=8,gres/gpu:3g.20gb=1,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,mem=64G              gres/gpu:3g.20gb=1 &lt;br /&gt;
gpuwide              2         2  1-00:00:00 cpu=8,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=1,mem=64G              gres/gpu:4g.20gb=1 &lt;br /&gt;
build               32         1    02:00:00 cpu=2,gres/gpu:3g.20gb=0,gres/gpu:40gb=0,gres/gpu:4g.20gb=0,gres/gpu=0,mem=16G&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The columns of this output are the following:&lt;br /&gt;
&lt;br /&gt;
:; Name&lt;br /&gt;
:: name of the QOS&lt;br /&gt;
&lt;br /&gt;
:; Priority&lt;br /&gt;
:: priority tier associated to the QOS (higher value = higher priority): see [[#Job priority|Job priority]] for details&lt;br /&gt;
&lt;br /&gt;
:; MaxSubmit&lt;br /&gt;
:: maximum number of jobs from a single user that can be submitted to SLURM with this QOS; submitted jobs include both running and queued jobs&lt;br /&gt;
:: See [[#Limits on jobs by the same user|Limits on jobs by the same user]] for an overview of the limits on jobs set by Mufasa.&lt;br /&gt;
&lt;br /&gt;
:; MaxWall&lt;br /&gt;
:: maximum wall clock duration of the jobs using the QOS (after which they are killed by SLURM), in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
:: For some QOSes these are not set: it means that they are determined by the [[#SLURM partitions|partition]]. Partitions also define the [[#Default values|default duration]] of jobs.&lt;br /&gt;
&lt;br /&gt;
:; MaxTRES&lt;br /&gt;
:: amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) available to a job using the QOS, where&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;cpu=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of CPUs (i.e., processor cores) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of CPUs specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;gres/&amp;#039;&amp;#039;gpu:Type&amp;#039;&amp;#039;=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum number of GPUs of class &amp;lt;code&amp;gt;&amp;#039;&amp;#039;Type&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; (see [[User Jobs#gres syntax|&amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax]]) is &amp;#039;&amp;#039;K&amp;#039;&amp;#039;&lt;br /&gt;
::: --&amp;gt; (for QOSes that allow access to GPUs) if not specified, the job cannot be launched&lt;br /&gt;
:: &amp;lt;code&amp;gt;&amp;#039;&amp;#039;&amp;#039;mem=&amp;#039;&amp;#039;K&amp;#039;&amp;#039;G&amp;#039;&amp;#039;&amp;#039;&amp;lt;/code&amp;gt; means that the maximum amount of system RAM is &amp;#039;&amp;#039;K&amp;#039;&amp;#039; GBytes&lt;br /&gt;
::: --&amp;gt; if not specified, the job gets the default amount of RAM specified by the [[#SLURM partitions|partition]]&lt;br /&gt;
&lt;br /&gt;
:; MinTRES&lt;br /&gt;
:: minimum amount of [[#System resources subjected to limitations|resources subjected to limitations]] (&amp;quot;&amp;#039;&amp;#039;Trackable RESources&amp;#039;&amp;#039;&amp;quot;) that a job using the QOS must request in order to actually get executed by SLURM.&lt;br /&gt;
:: (If your job does not actually need these resources, you&amp;#039;ve chosen the wrong QOS: you can use one with a higher priority.)&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; QOS is the default one, and exists only to ensure that users always specify a QOS when running a job. Since &amp;lt;code&amp;gt;normal&amp;lt;/code&amp;gt; has zero priority and no resources, a job run using this QOS would never be run.&lt;br /&gt;
&lt;br /&gt;
The information provided by the &amp;lt;code&amp;gt;sacctmgr list qos&amp;lt;/code&amp;gt; command above is summarised by the following table:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! QOS&lt;br /&gt;
! Priority&amp;lt;/br&amp;gt;tier&lt;br /&gt;
! Max&amp;lt;/br&amp;gt;Submit&lt;br /&gt;
! MaxWall&amp;lt;/br&amp;gt;[h]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;CPUs&lt;br /&gt;
! max RAM&amp;lt;/br&amp;gt;[GB]&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;3g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;4g.20GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
! max # of&amp;lt;/br&amp;gt;40GB&amp;lt;/br&amp;gt;GPUs&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | build&lt;br /&gt;
| 32&lt;br /&gt;
| 1&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 16&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpulight&lt;br /&gt;
| 8&lt;br /&gt;
| 1&lt;br /&gt;
| 12&lt;br /&gt;
| 2&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | nogpu&lt;br /&gt;
| 4&lt;br /&gt;
| 1&lt;br /&gt;
| 72&lt;br /&gt;
| 16&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpu&lt;br /&gt;
| 2&lt;br /&gt;
| 1&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuwide&lt;br /&gt;
| 2&lt;br /&gt;
| 2&lt;br /&gt;
| 24&lt;br /&gt;
| 8&lt;br /&gt;
| 64&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-20&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| 2 (min=1)&lt;br /&gt;
| -&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | gpuheavy-40&lt;br /&gt;
| 1&lt;br /&gt;
| 1&lt;br /&gt;
| 72 (set by partition)&lt;br /&gt;
| 8&lt;br /&gt;
| 128&lt;br /&gt;
| -&lt;br /&gt;
| -&lt;br /&gt;
| 1 (min=1)&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
A key piece of information in the table above is the &amp;#039;&amp;#039;&amp;#039;priority tier&amp;#039;&amp;#039;&amp;#039; associated to each QOS. In Mufasa 2.0, priority tiers are used to encourage users to use the &amp;#039;&amp;#039;&amp;#039;least powerful QOS that is compatible with their needs&amp;#039;&amp;#039;&amp;#039;, where &amp;quot;powerful&amp;quot; means &amp;quot;rich with resources&amp;quot;. Less powerful QOSes increase the priority of the jobs that use them, so these jobs tend to be executed sooner. See [[#Job priority|Job priority]] for details about how priority affects the execution order of jobs in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
== The &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS ==&lt;br /&gt;
&lt;br /&gt;
This QOS is specifically designed to be used by Mufasa users to &amp;#039;&amp;#039;&amp;#039;quickly build [[System#Containers|container images]]&amp;#039;&amp;#039;&amp;#039;. Its associated priority tier is very high: this, combined with the fact that jobs using this QOS require few resources, means that such jobs usually get executed very soon.&lt;br /&gt;
&lt;br /&gt;
On the other hand, the limited resources and the lack of access to the GPUs make the &amp;lt;code&amp;gt;build&amp;lt;/code&amp;gt; QOS unsuitable for other tasks.&lt;br /&gt;
&lt;br /&gt;
See [[Singularity#Building Singularity images|Building Singularity images]] for directions about building Singularity container images.&lt;br /&gt;
&lt;br /&gt;
== Restricted QOSes ==&lt;br /&gt;
&lt;br /&gt;
In Mufasa, the most powerful QOSes are reserved to researchers (called &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users: these include academic personnel and Ph.D. students). M.Sc. students, called &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users, cannot use them while running jobs.&lt;br /&gt;
&lt;br /&gt;
See [[#research users and students users|below]] to understand how &amp;#039;&amp;#039;user categories&amp;#039;&amp;#039; work in Mufasa 2.0.&lt;br /&gt;
&lt;br /&gt;
= &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users =&lt;br /&gt;
&lt;br /&gt;
Users of Mufasa belong to two &amp;#039;&amp;#039;&amp;#039;user categories&amp;#039;&amp;#039;&amp;#039;. Categories provide users with different access to  to Mufasa&amp;#039;s resources: the idea being to provide researchers with more access while still letting students benefit from the server&amp;#039;s capabilities.&lt;br /&gt;
&lt;br /&gt;
User categories in Mufasa are the following:&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. academic personnel and Ph.D. students&lt;br /&gt;
::: * can use all QOSes, including the the [[#Restricted QOSes|restricted ones]]&lt;br /&gt;
::: * their jobs have a higher &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a higher number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
:: &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;, i.e. M.Sc. students&lt;br /&gt;
::: * cannot use the [[#Restricted QOSes|restricted QOSes]]&lt;br /&gt;
::: * their jobs have a lower &amp;#039;&amp;#039;base priority&amp;#039;&amp;#039;&lt;br /&gt;
::: * have a lower number of jobs that can be running at the same time&lt;br /&gt;
&lt;br /&gt;
You can inspect the differences in priority and running jobs between &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users with command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association format=&amp;quot;account,priority,maxjobs&amp;quot; | grep -E &amp;#039;Account|research|students&amp;#039;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
   Account   Priority MaxJobs &lt;br /&gt;
  research          4       2 &lt;br /&gt;
  students          1       1&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
To know what limits apply to your own user, use command&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sacctmgr list association where user=$USER format=&amp;quot;user,priority,maxjobs,qos%-60&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
      User   Priority MaxJobs QOS                                                          &lt;br /&gt;
---------- ---------- ------- ------------------------------------------------------------ &lt;br /&gt;
    preali          4       2 build,gpu,gpuheavy-20,gpuheavy-40,gpulight,gpuwide,nogpu&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The list under &amp;quot;QOS&amp;quot; shows what QOSes your user is allowed to use when [[User Jobs|running jobs]]. &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; users can use all of them, while &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; users can only access a subset of them.&lt;br /&gt;
&lt;br /&gt;
= Limits on jobs by the same user =&lt;br /&gt;
&lt;br /&gt;
Mufasa enforces limits on the number of jobs from a single user. Such limits aim at preventing users from &amp;quot;hogging&amp;quot; system resources, and apply to:&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;submitted jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that the user asked SLURM to execute, each of which may currently be either running or queued&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;running jobs&amp;#039;&amp;#039;&amp;#039;, i.e. jobs that are currently in execution&lt;br /&gt;
&lt;br /&gt;
Limits on submitted jobs are set via [[#SLURM Quality of Service (QOS)|QOSes]], while limits on running jobs are set via [[#research users and students users|user category]].&lt;br /&gt;
&lt;br /&gt;
The following table summarises what limits exist:&lt;br /&gt;
&lt;br /&gt;
{| class=&amp;quot;wikitable&amp;quot; style=&amp;quot;text-align:center;&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! &lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;running&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
! on the number of &amp;lt;u&amp;gt;submitted&amp;lt;/u&amp;gt; jobs&amp;lt;/br&amp;gt;by a single user&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | global limits&amp;lt;br/&amp;gt;(system-wide)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for&amp;#039;&amp;#039; &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;users&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to limits set by individual QOSes (below)&lt;br /&gt;
|-&lt;br /&gt;
! rowspan=&amp;quot;1&amp;quot; style=&amp;quot;text-align:center;&amp;quot; | limits for&amp;lt;br/&amp;gt;individual&amp;lt;/br&amp;gt;QOSes&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;not limited directly...&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;...but subject to global limits (above)&lt;br /&gt;
| &amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;2 for the&amp;#039;&amp;#039; &amp;lt;code&amp;gt;gpuwide&amp;lt;/code&amp;gt; &amp;#039;&amp;#039;QOS&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;lt;br/&amp;gt;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;1 for each of the other QOSes&amp;#039;&amp;#039;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
= Job priority =&lt;br /&gt;
&lt;br /&gt;
Once the execution of a job has been requested, the job is not run immediately: it is instead &amp;#039;&amp;#039;queued&amp;#039;&amp;#039; by SLURM, together with all the other jobs awaiting execution. The job on top of the queue at any time is the first to be put into execution as soon as the resources it requires are available. &lt;br /&gt;
&lt;br /&gt;
The order of the items in the job queue depends on their &amp;#039;&amp;#039;&amp;#039;priority&amp;#039;&amp;#039;&amp;#039;. The job with the highest priority is put by SLURM on top of the queue, while all other jobs are put in the queue in order of descending priority.&lt;br /&gt;
&lt;br /&gt;
The goal of SLURM is to maximise resource availability: i.e., to ensure the shorter possible wait time before job execution. To achieve this goal, in Mufasa SLURM is configured to &amp;#039;&amp;#039;&amp;#039;encourage users not to ask for resources or execution time that their job doesn&amp;#039;t need&amp;#039;&amp;#039;&amp;#039;. This is done via the priority mechanism: the more resources and/or time a job requests, the lower its priority will be; and the later it will be executed.&lt;br /&gt;
&lt;br /&gt;
Priority management in Mufasa is designed to set up a &amp;#039;&amp;#039;&amp;#039;virtuous cycle&amp;#039;&amp;#039;&amp;#039; where users, by carefully choosing what to ask for, obtain two results:&lt;br /&gt;
* they ensure that their job is executed as soon as possible;&lt;br /&gt;
* they leave as much as possible of Mufasa&amp;#039;s resources free for other users&amp;#039;s jobs.&lt;br /&gt;
&lt;br /&gt;
== Elements determining job priority ==&lt;br /&gt;
In Mufasa, the priority of a job is computed by SLURM according to the following elements:&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#research users and students users|User category]]&amp;#039;&amp;#039;&amp;#039; (i.e., &amp;lt;code&amp;gt;research&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;students&amp;lt;/code&amp;gt;)&lt;br /&gt;
::: Used to provide higher priority to jobs run by &amp;#039;&amp;#039;&amp;#039;research personnel&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;[[#SLURM Quality of Service (QOS)|QOS]]&amp;#039;&amp;#039;&amp;#039; used by the job&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;less resources&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Number of CPUs&amp;#039;&amp;#039;&amp;#039; requested by the job (also called &amp;quot;job size&amp;quot;)&lt;br /&gt;
::: Used to provide higher priority to jobs asking for &amp;#039;&amp;#039;&amp;#039;a lower number of CPUs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job duration&amp;#039;&amp;#039;&amp;#039;, i.e. the execution time requested by the job&lt;br /&gt;
::: Used to provide higher priority to &amp;#039;&amp;#039;&amp;#039;shorter jobs&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;Job Age&amp;#039;&amp;#039;&amp;#039;, i.e. the time that the job has been waiting in the queue&lt;br /&gt;
::: Used to provide higher priority to jobs which have been &amp;#039;&amp;#039;&amp;#039;queued for a longer time&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
: &amp;#039;&amp;#039;&amp;#039;FairShare&amp;#039;&amp;#039;&amp;#039;, i.e. a factor computed by SLURM to balance use of the system by different users&lt;br /&gt;
::: Used to provide higher priority to jobs by users who &amp;#039;&amp;#039;&amp;#039;used Mufasa&amp;#039;s resources less than others&amp;#039;&amp;#039;&amp;#039; (see [[#How FairShare works|below]])&lt;br /&gt;
&lt;br /&gt;
You can understand how the priority of your jobs depends on each of these factors using command&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sprio&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
Here is an example of the output of &amp;lt;code&amp;gt;sprio&amp;lt;/code&amp;gt;:&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
          JOBID PARTITION   PRIORITY       SITE        AGE      ASSOC  FAIRSHARE    JOBSIZE        QOS&lt;br /&gt;
           5193 jobs            5987          0        871       2500         39       2499         78&lt;br /&gt;
           5201 jobs            4275          0        152        625        844       2498        156&lt;br /&gt;
           5202 jobs            4123          0          0        625        844       2498        156&lt;br /&gt;
           5203 jobs            5533          0        131       2500        325       2499         78&lt;br /&gt;
           5204 jobs            4052          0        124        625        649       2498        156&lt;br /&gt;
           5205 jobs            5494          0        107       2500        312       2498         78&lt;br /&gt;
           5212 jobs            4165          0         32        625        857       2495        156&lt;br /&gt;
           5231 jobs            5181          0          3       2500         26       2497        156&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
=== How FairShare works ===&lt;br /&gt;
In Mufasa 2.0, the FairShare of each user is computed according to these rules:&lt;br /&gt;
# It considers the quantity used by (current and past) user jobs of:&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;CPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of CPUs used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;RAM&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the amount of RAM used&lt;br /&gt;
#* &amp;#039;&amp;#039;&amp;#039;GPUs&amp;#039;&amp;#039;&amp;#039; - impact on FairShare is proportional to the number of GPUs used&lt;br /&gt;
#** note: 40 GB GPUs have double the impact on FairShare than 20 GB GPUs&lt;br /&gt;
# The impact on FairShare of any use of resources is &amp;#039;&amp;#039;&amp;#039;proportional to its duration&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
#* example: using 32 GB of RAM for 48h has the same impact of using 64 GB of RAM for 24h&lt;br /&gt;
# FairShare has a &amp;quot;fading memory&amp;quot;: i.e., resource use has less and less impact on FairShare the farther in the past it is&lt;br /&gt;
#* consequence: over time, the &amp;quot;history&amp;quot; of a user gets forgotten by FairShare&lt;br /&gt;
&lt;br /&gt;
== How to maximise the priority of your jobs ==&lt;br /&gt;
&lt;br /&gt;
Every time you run a SLURM job, follow these guidelines:&lt;br /&gt;
&lt;br /&gt;
:{|class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|&lt;br /&gt;
; Choose the less powerful QOS compatible with the needs of your job&lt;br /&gt;
:: QOSes with access to less resources have higher base priority&lt;br /&gt;
&lt;br /&gt;
; Only request CPUs and RAM that your job really needs&lt;br /&gt;
:: Usually, code exploits multiple CPUs only if specifically designed to do so: if unsure, only ask for 1 CPU&lt;br /&gt;
:: The fewer resources your job asks for, the less it will wait before they become available&lt;br /&gt;
:: Asking for fewer CPUs and/or less RAM improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Do not request more time than your jobs needs to complete&lt;br /&gt;
:: Make a worst-case estimate and only ask for that duration&lt;br /&gt;
:: Asking for less of Mufasa&amp;#039;s time improves your FairShare&lt;br /&gt;
&lt;br /&gt;
; Debug and test code using less powerful QOSes before running it with more powerful QOSes&lt;br /&gt;
:: Your test jobs will get a higher priority and your FairShare will improve&lt;br /&gt;
&lt;br /&gt;
; Cancel jobs that you don&amp;#039;t need them anymore&lt;br /&gt;
:: Always use [[User_Jobs#Cancelling_a_job_with_scancel|scancel]] to delete completed (or crashed) jobs: your Fairshare will improve&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
Protip: it&amp;#039;s a good idea to [[User Jobs#Looking for unused GPUs|check for unused GPUs]] before choosing what to request. Requesting a GPU that is currently idle will help your job get executed sooner.&lt;br /&gt;
&lt;br /&gt;
= SLURM partitions =&lt;br /&gt;
&lt;br /&gt;
Partitions are another mechanism provided by SLURM to create different levels of access to system resources. Since in Mufasa 2.0 access to resources is controlled via [[#SLURM Quality of Service (QOS)|QOSes]], partitions are not very relevant. (This is one of the main differences between Mufasa 1.0 and Mufasa 2.0.)&lt;br /&gt;
&lt;br /&gt;
Note, however, that the default values for some features of SLURM jobs (e.g., duration) are [[#Default values|set by the partition]].&lt;br /&gt;
&lt;br /&gt;
In Mufasa 2.0, there is a single SLURM partition, called &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt;, and all jobs run on it. The state of &amp;lt;code&amp;gt;jobs&amp;lt;/code&amp;gt; can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
sinfo -o &amp;quot;%10P %5a %9T %11L %10l&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PARTITION  AVAIL STATE     DEFAULTTIME TIMELIMIT &lt;br /&gt;
jobs*      up    idle      1:00:00     3-00:00:00&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
where columns correspond to the following information:&lt;br /&gt;
&lt;br /&gt;
:; PARTITION&lt;br /&gt;
:: name of the partition; the asterisks indicates that it&amp;#039;s the default one&lt;br /&gt;
&lt;br /&gt;
:; AVAIL&lt;br /&gt;
:: state/availability of the partition: see [[#Partition availability|below]]&lt;br /&gt;
&lt;br /&gt;
:; STATE&lt;br /&gt;
:: state (using [https://slurm.schedmd.com/sinfo.html#SECTION_NODE-STATE-CODES these codes])&lt;br /&gt;
:: typical values are &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;mixed&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that some of the resources are busy executing jobs while other are idle, and &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;allocated&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; - meaning that all of the resources are in use&lt;br /&gt;
&lt;br /&gt;
:; DEFAULTTIME&lt;br /&gt;
:: default runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
:; TIMELIMIT&lt;br /&gt;
:: maximum runtime of a job, in format &amp;#039;&amp;#039;[days-]hours:minutes:seconds&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
The asterisk at the end of the partition name indicates the default partition, i.e. the one on which jobs which do not ask for a specific partition are run.&lt;br /&gt;
&lt;br /&gt;
Command &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; does not tell you about the &amp;#039;&amp;#039;jobs&amp;#039;&amp;#039; submitted to a partition. This information is obtained, instead, with [[User Jobs#Inspecting jobs with squeue|command &amp;lt;code&amp;gt;squeue&amp;lt;/code&amp;gt;]].&lt;br /&gt;
&lt;br /&gt;
== Partition availability ==&lt;br /&gt;
&lt;br /&gt;
The most important information that &amp;lt;code&amp;gt;sinfo&amp;lt;/code&amp;gt; provides is the &amp;#039;&amp;#039;&amp;#039;availability&amp;#039;&amp;#039;&amp;#039; (also called &amp;#039;&amp;#039;&amp;#039;state&amp;#039;&amp;#039;&amp;#039;) of partitions. This is shown in column &amp;quot;AVAIL&amp;quot;. Possible partition states are:&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is available&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed as soon as resources allow&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is in the process of becoming unavailable (i.e., of entering the &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; state: see below)&lt;br /&gt;
:: Currently running jobs will be completed&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
:&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; = the partition is unavailable&lt;br /&gt;
:: There are no running jobs&lt;br /&gt;
:: It&amp;#039;s not possible to launch jobs on the partition&lt;br /&gt;
:: Queued jobs will be executed when the partition becomes available again (i.e. goes back to the &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; state)&lt;br /&gt;
&lt;br /&gt;
When a partition goes from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; no harm is done to running jobs. In a normally functioning SLURM system, the passage from &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; to &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; happens only when no jobs are running on the partition. If (e.g., due to a malfunction) the passage happens with jobs still running, they get killed.&lt;br /&gt;
&lt;br /&gt;
A partition in state &amp;lt;code&amp;gt;drain&amp;lt;/code&amp;gt; or &amp;lt;code&amp;gt;down&amp;lt;/code&amp;gt; requires intervention by a [[Roles|Job Administrator]] to be restored to &amp;lt;code&amp;gt;up&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
== Default values ==&lt;br /&gt;
&lt;br /&gt;
The features of SLURM partitions, including the &amp;#039;&amp;#039;&amp;#039;default values&amp;#039;&amp;#039;&amp;#039; which are applied to jobs that do not make explicit requests, can be inspected with&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
scontrol show partition&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
which provides an output similar to this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre style=&amp;quot;color: lightgrey; background: black;&amp;quot;&amp;gt;&lt;br /&gt;
PartitionName=jobs&lt;br /&gt;
   AllowGroups=ALL AllowAccounts=ALL AllowQos=nogpu,gpulight,gpu,gpuwide,gpuheavy-20,gpuheavy-40&lt;br /&gt;
   AllocNodes=ALL Default=YES QoS=N/A&lt;br /&gt;
=&amp;gt; DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO&lt;br /&gt;
   MaxNodes=UNLIMITED MaxTime=3-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED&lt;br /&gt;
   Nodes=gn01&lt;br /&gt;
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO&lt;br /&gt;
   OverTimeLimit=NONE PreemptMode=OFF&lt;br /&gt;
   State=UP TotalCPUs=48 TotalNodes=1 SelectTypeParameters=NONE&lt;br /&gt;
   JobDefaults=(null)&lt;br /&gt;
=&amp;gt; DefMemPerNode=4096 MaxMemPerNode=UNLIMITED&lt;br /&gt;
   TRES=cpu=48,mem=1011435M,node=1,billing=49,gres/gpu=13,gres/gpu:3g.20gb=5,gres/gpu:40gb=3,gres/gpu:4g.20gb=5&lt;br /&gt;
   TRESBillingWeights=cpu=1.0,gres/gpu:3g.20gb=6.0,gres/gpu:4g.20gb=6.0,gres/gpu:40gb=6.0,mem=0.05g&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In the example, we have highlighted with  &amp;#039;&amp;#039;&amp;#039;=&amp;gt;&amp;#039;&amp;#039;&amp;#039;  the most relevant default values for Mufasa users, i.e.:&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefaultTime&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default execution time assigned to a job run on the partition (e.g., 1 hour)&lt;br /&gt;
&lt;br /&gt;
;&amp;lt;code&amp;gt;DefMemPerNode&amp;lt;/code&amp;gt;&lt;br /&gt;
:: the default amount of RAM assigned to a job run on the partition (e.g., 4GB)&lt;br /&gt;
&lt;br /&gt;
= System resources subjected to limitations =&lt;br /&gt;
&lt;br /&gt;
In systems based on SLURM like Mufasa, &amp;#039;&amp;#039;&amp;#039;TRES (Trackable RESources)&amp;#039;&amp;#039;&amp;#039; are (from [https://slurm.schedmd.com/tres.html SLURM&amp;#039;s documentation] &amp;quot;&amp;#039;&amp;#039;resources that can be tracked for usage or used to enforce limits against.&amp;#039;&amp;#039;&amp;quot;&lt;br /&gt;
&lt;br /&gt;
TRES include CPUs, RAM and &amp;#039;&amp;#039;&amp;#039;GRES&amp;#039;&amp;#039;&amp;#039;. The last term stands for &amp;#039;&amp;#039;Generic RESources&amp;#039;&amp;#039; that a job may need for its execution. In Mufasa, the only &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; resources are the GPUs.&lt;br /&gt;
&lt;br /&gt;
== &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; syntax ==&lt;br /&gt;
&lt;br /&gt;
To ask SLURM to assign GRES resources (i.e., GPUs) to a job, a special syntax must be used. Precisely, the name of each GPU resource takes the form&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:Name:Type&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
Considering the [[System#CPUs and GPUs|GPU complement of Mufasa]], &amp;lt;code&amp;gt;Type&amp;lt;/code&amp;gt; takes the following values:&lt;br /&gt;
&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:40gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 40 Gbytes of RAM&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:4g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 4 compute units&lt;br /&gt;
* &amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039; for GPUs with 20 Gbytes of RAM and 3 compute units&lt;br /&gt;
&lt;br /&gt;
So, for instance,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:3g.20gb&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
identifies a resource corresponding to a GPU with 20 GB of RAM and 3 compute units.&lt;br /&gt;
&lt;br /&gt;
When asking for a GRES resource (e.g., in an &amp;lt;code&amp;gt;srun&amp;lt;/code&amp;gt; command or an &amp;lt;code&amp;gt;SBATCH&amp;lt;/code&amp;gt; directive of an [[User Jobs#Using execution scripts to run jobs|execution script]]), the syntax required by SLURM is&lt;br /&gt;
&lt;br /&gt;
&amp;#039;&amp;#039;&amp;#039;&amp;lt;code&amp;gt;gpu:&amp;lt;Type&amp;gt;:&amp;lt;Quantity&amp;gt;&amp;lt;/code&amp;gt;&amp;#039;&amp;#039;&amp;#039;&lt;br /&gt;
&lt;br /&gt;
where &amp;lt;code&amp;gt;Quantity&amp;lt;/code&amp;gt; is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of type &amp;lt;code&amp;gt;4g.20gb&amp;lt;/code&amp;gt; the syntax is&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;gpu:4g.20gb:2&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SLURM&amp;#039;s &amp;#039;&amp;#039;generic resources&amp;#039;&amp;#039; are defined in &amp;lt;code&amp;gt;/etc/slurm/gres.conf&amp;lt;/code&amp;gt;. In order to make GPUs available to SLURM&amp;#039;s &amp;lt;code&amp;gt;gres&amp;lt;/code&amp;gt; management, Mufasa makes use of Nvidia&amp;#039;s [https://developer.nvidia.com/nvidia-management-library-nvml NVML library]. For additional information see [https://slurm.schedmd.com/gres.html SLURM&amp;#039;s documentation].&lt;/div&gt;</summary>
		<author><name>GiulioFontana</name></author>
	</entry>
</feed>