Mufasa (BioHPC) - Recent changes [en]

System

GiulioFontana — Mon, 08 Jun 2026 09:54:04 GMT

How to find out who is using a shared storage space

← Older revision		Revision as of 09:54, 8 June 2026
(6 intermediate revisions by the same user not shown)
Line 237:		Line 237:
	::: The general rule is: ''keep in your home directory '''only''' the files that the work you are doing on Mufasa '''right now''' needs; remove a file as soon as it is not needed anymore for your current work''.		::: The general rule is: ''keep in your home directory '''only''' the files that the work you are doing on Mufasa '''right now''' needs; remove a file as soon as it is not needed anymore for your current work''.
	::: ''Mufasa is not a storage space!''		::: ''Mufasa is not a storage space!''

			To know how space is available in your personal directory and how much of that space is currently in use, you can use command

			<pre style="color: lightgrey; background: black;">
			df -h ~
			</pre>

	:; Shared directories		:; Shared directories
Line 251:		Line 257:
	::: - For '''faster read/write'''. Shared spaces are physically located on faster disks wrt the personal home directories (SSDs instead of mechanical HDDs). When a processing job requires reading or writing very large amounts of data, placing such data in a shared directory can significantly speed up the job.		::: - For '''faster read/write'''. Shared spaces are physically located on faster disks wrt the personal home directories (SSDs instead of mechanical HDDs). When a processing job requires reading or writing very large amounts of data, placing such data in a shared directory can significantly speed up the job.
	::: '''Important!''' Shared directories are used by several people, so it's important to ''quickly remove from them any file that is not actively in use''.		::: '''Important!''' Shared directories are used by several people, so it's important to ''quickly remove from them any file that is not actively in use''.

			To know how space is available in a shared directory and how much of such space is currently in use, you can use command

			<pre style="color: lightgrey; background: black;">
			df -h <path of shared directory>
			</pre>

			=== How to find out who is using a shared storage space ===
			If you have access to a shared directory, you can use

			<pre style="color: lightgrey; background: black;">
			ls -l <path to shared directory>
			</pre>

			to see a list of its contents with size and owner of each item.

			This allows you to know who to ask if you need someone to delete some some files to make space for other users. Shared directories are shared spaces, and files should never be left there for longer than strictly necessary; however, sometimes one forgets :-)

	== Disk quotas ==		== Disk quotas ==

User Jobs

GiulioFontana — Wed, 27 May 2026 11:54:01 GMT

← Older revision		Revision as of 11:54, 27 May 2026
Line 359:		Line 359:
	== Cancelling completed jobs ==		== Cancelling completed jobs ==

	When a user process run via SLURM has completed its execution and is not needed anymore, it is important to [[~~User_Jobs~~#~~Canceling_a_job_with_scancel~~\|close it with scancel]]. Especially if much time remains to the end of the execution time requested by the job.		When a user process run via SLURM has completed its execution and is not needed anymore, it is important to [[#Cancelling a job with scancel\|close it with scancel]]. Especially if much time remains to the end of the execution time requested by the job.

	Cancelling a SLURM job makes the resources reserved by SLURM free again for other users, and thus speeds up the execution of the jobs still queued.		Cancelling a SLURM job makes the resources reserved by SLURM free again for other users, and thus speeds up the execution of the jobs still queued.

SLURM

GiulioFontana — Tue, 26 May 2026 11:36:12 GMT

How to maximise the priority of your jobs

← Older revision		Revision as of 11:36, 26 May 2026
(15 intermediate revisions by the same user not shown)
Line 262:		Line 262:
	\|-		\|-
	! rowspan="1" style="text-align:center;" \| limits for<br/>individual</br>QOSes		! rowspan="1" style="text-align:center;" \| limits for<br/>individual</br>QOSes
	\| '''''not limited directly...'''''<br/>...but subject to global limits ~~on running jobs~~ (above~~)<br/>and to limits set by individual QOSes (on the right~~)		\| '''''not limited directly...'''''<br/>...but subject to global limits (above)
	\| '''''2 for the'' <code>gpuwide</code> ''QOS'''''<br/>'''''1 for each of the other QOSes'''''		\| '''''2 for the'' <code>gpuwide</code> ''QOS'''''<br/>'''''1 for each of the other QOSes'''''
	\|}		\|}
Line 318:		Line 318:
	\|		\|
	; Choose the less powerful QOS compatible with the needs of your job		; Choose the less powerful QOS compatible with the needs of your job
	:: QOSes with access to less resources ~~lead to~~ higher priority		:: QOSes with access to less resources have higher base priority

	; Only request CPUs and RAM that your job ~~will actually use~~		; Only request CPUs and RAM that your job really needs
	:: ~~If you didn't design your~~ code ~~to exploit~~ multiple CPUs~~, check that it does! If it doesn't~~, only ask for 1 CPU		:: Usually, code exploits multiple CPUs only if designed to do so: if unsure, only ask for 1 CPU
	:: ~~If you ask for more RAM than needed,~~ your job ~~has to wait~~ for it ~~to be~~ available, and ~~you lower~~ your FairShare		:: The fewer resources your job asks for, the less it will wait before they become available
			:: Asking for fewer CPUs and/or less RAM improves your FairShare

	; Do not request more time than your jobs needs to complete		; Do not request more time than your jobs needs to complete
	:: Make a worst-case estimate and only ask for that duration		:: Make a worst-case estimate and only ask for that duration
			:: Asking for less of Mufasa's time improves your FairShare

	; ~~Test~~ and ~~debug your~~ code using less powerful QOSes before running it on more powerful QOSes		; Debug and test code using less powerful QOSes before running it with more powerful QOSes
	:: Your test jobs will get a higher priority and your FairShare will improve		:: Your test jobs will get a higher priority and your FairShare will improve

	; Cancel jobs ~~when~~ you don't need them anymore		; Cancel jobs that you don't need them anymore
	:: ~~Use~~ [[User_Jobs#Cancelling_a_job_with_scancel\|scancel]] to delete ~~your jobs when finished~~ (or if crashed ~~due to a bug~~): your Fairshare will improve		:: Always use [[User_Jobs#Cancelling_a_job_with_scancel\|scancel]] to delete completed (or crashed) jobs: your Fairshare will improve
	\|}		\|}

	~~Suggestion~~: ~~if you're going to run a job,~~ it's a good idea to [[#Looking for unused GPUs\|~~look~~ for unused GPUs]] before choosing what ~~GPU~~ to request. ~~Choosing~~ a GPU that is currently idle ~~should~~ help your job get ~~run~~ sooner.		Protip: it's a good idea to [[User Jobs#Looking for unused GPUs\|check for unused GPUs]] before choosing what to request. Requesting a GPU that is currently idle will help your job get executed sooner.

	~~= System resources subjected to limitations =~~

	In systems based on SLURM like Mufasa, '''TRES (Trackable RESources)''' are (from [https://slurm.schedmd.com/tres.html SLURM's documentation] "''resources that can be tracked for usage or used to enforce limits against.''"

	~~TRES include CPUs, RAM and '''GRES'''. The last term stands for ''Generic RESources'' that a job may need for its execution. In Mufasa, the only <code>gres</code> resources are the GPUs.~~

	~~== <code>gres</code> syntax ==~~

	~~To ask SLURM to assign GRES resources (i.e., GPUs) to a job, a special syntax must be used. Precisely, the name of each GPU resource takes the form~~

	~~'''<code>gpu:Name:Type</code>'''~~

	~~Considering the [[System#CPUs and GPUs\|GPU complement of Mufasa]], <code>Type</code> takes the following values:~~

	* '''<code>gpu:40gb</code>''' for GPUs with 40 Gbytes of RAM
	* '''<code>gpu:4g.20gb</code>''' for GPUs with 20 Gbytes of RAM and 4 compute units
	* '''<code>gpu:3g.20gb</code>''' for GPUs with 20 Gbytes of RAM and 3 compute units

	~~So, for instance,~~

	~~<code>gpu:3g.20gb</code>~~

	~~identifies a resource corresponding to a GPU with 20 GB of RAM and 3 compute units.~~

	When asking for a GRES resource (e.g., in an <code>srun</code> command or an <code>SBATCH</code> directive of an [[User Jobs#Using execution scripts to run jobs\|execution script]]), the syntax required by SLURM is

	~~'''<code>gpu:<Type>:<Quantity></code>'''~~

	~~where <code>Quantity</code> is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of type <code>4g.20gb</code> the syntax is~~

	~~<code>gpu:4g.20gb:2</code>~~

	SLURM's ''generic resources'' are defined in <code>/etc/slurm/gres.conf</code>. In order to make GPUs available to SLURM's <code>gres</code> management, Mufasa makes use of Nvidia's [https://developer.nvidia.com/nvidia-management-library-nvml NVML library]. For additional information see [https://slurm.schedmd.com/gres.html SLURM's documentation].

	~~== Looking for unused GPUs ==~~

	GPUs are usually the most limited resource on Mufasa. So, if your job requires a GPU, the best way to get it executed quickly is to use a QOS associated to a type of GPU of which there are one or more that aren't currently in use. This command

	~~<pre style="color: lightgrey; background: black;">~~
	~~sinfo -O Gres:100~~
	~~</pre>~~

	~~provides a summary of all the Gres (i.e., GPU) resources possessed by Mufasa. It provides an output similar to the following:~~

	~~<pre style="color: lightgrey; background: black;">~~
	~~GRES~~
	~~gpu:40gb:3,gpu:4g.20gb:5,gpu:3g.20gb:5~~
	~~</pre>~~

	~~To know which of the GPUs are currently in use, use command~~

	~~<pre style="color: lightgrey; background: black;">~~
	~~sinfo -O GresUsed:100~~
	~~</pre>~~

	~~which provides an output similar to this:~~

	~~<pre style="color: lightgrey; background: black;">~~
	~~GRES_USED~~
	~~gpu:40gb:2(IDX:0-1),gpu:4g.20gb:2(IDX:5,8),gpu:3g.20gb:3(IDX:3-4,6)~~
	~~</pre>~~

	~~By comparing the two lists (GRES and GRES_USED) you can easily spot unused GPUs~~.

	= SLURM partitions =		= SLURM partitions =
Line 496:		Line 434:
	;<code>DefMemPerNode</code>		;<code>DefMemPerNode</code>
	:: the default amount of RAM assigned to a job run on the partition (e.g., 4GB)		:: the default amount of RAM assigned to a job run on the partition (e.g., 4GB)

			= System resources subjected to limitations =

			In systems based on SLURM like Mufasa, '''TRES (Trackable RESources)''' are (from [https://slurm.schedmd.com/tres.html SLURM's documentation] "''resources that can be tracked for usage or used to enforce limits against.''"

			TRES include CPUs, RAM and '''GRES'''. The last term stands for ''Generic RESources'' that a job may need for its execution. In Mufasa, the only <code>gres</code> resources are the GPUs.

			== <code>gres</code> syntax ==

			To ask SLURM to assign GRES resources (i.e., GPUs) to a job, a special syntax must be used. Precisely, the name of each GPU resource takes the form

			'''<code>gpu:Name:Type</code>'''

			Considering the [[System#CPUs and GPUs\|GPU complement of Mufasa]], <code>Type</code> takes the following values:

			* '''<code>gpu:40gb</code>''' for GPUs with 40 Gbytes of RAM
			* '''<code>gpu:4g.20gb</code>''' for GPUs with 20 Gbytes of RAM and 4 compute units
			* '''<code>gpu:3g.20gb</code>''' for GPUs with 20 Gbytes of RAM and 3 compute units

			So, for instance,

			<code>gpu:3g.20gb</code>

			identifies a resource corresponding to a GPU with 20 GB of RAM and 3 compute units.

			When asking for a GRES resource (e.g., in an <code>srun</code> command or an <code>SBATCH</code> directive of an [[User Jobs#Using execution scripts to run jobs\|execution script]]), the syntax required by SLURM is

			'''<code>gpu:<Type>:<Quantity></code>'''

			where <code>Quantity</code> is an integer value specifying how many items of the resource are requested. So, for instance, to ask for 2 GPUs of type <code>4g.20gb</code> the syntax is

			<code>gpu:4g.20gb:2</code>

			SLURM's ''generic resources'' are defined in <code>/etc/slurm/gres.conf</code>. In order to make GPUs available to SLURM's <code>gres</code> management, Mufasa makes use of Nvidia's [https://developer.nvidia.com/nvidia-management-library-nvml NVML library]. For additional information see [https://slurm.schedmd.com/gres.html SLURM's documentation].

User Jobs

GiulioFontana — Tue, 26 May 2026 11:13:58 GMT

← Older revision		Revision as of 11:13, 26 May 2026
Line 366:		Line 366:

	Please note that [[System#Job priority\|job priority]] for your user depends (also) on the overall duration of the jobs that you ran on Mufasa. Therefore, '''cancelling jobs that are not needed anymore improves your future jobs' priority'''.		Please note that [[System#Job priority\|job priority]] for your user depends (also) on the overall duration of the jobs that you ran on Mufasa. Therefore, '''cancelling jobs that are not needed anymore improves your future jobs' priority'''.

			= Looking for unused GPUs =

			GPUs are usually the most limited resource on Mufasa. So, if your job requires a GPU, the best way to get it executed quickly is to use a QOS associated to a type of GPU of which there are one or more that aren't currently in use. This command

			<pre style="color: lightgrey; background: black;">
			sinfo -O Gres:100
			</pre>

			provides a summary of all the Gres (i.e., GPU) resources possessed by Mufasa. It provides an output similar to the following:

			<pre style="color: lightgrey; background: black;">
			GRES
			gpu:40gb:3,gpu:4g.20gb:5,gpu:3g.20gb:5
			</pre>

			To know which of the GPUs are currently in use, use command

			<pre style="color: lightgrey; background: black;">
			sinfo -O GresUsed:100
			</pre>

			which provides an output similar to this:

			<pre style="color: lightgrey; background: black;">
			GRES_USED
			gpu:40gb:2(IDX:0-1),gpu:4g.20gb:2(IDX:5,8),gpu:3g.20gb:3(IDX:3-4,6)
			</pre>

			By comparing the two lists (GRES and GRES_USED) you can easily spot unused GPUs.

	= Detaching from a running job with <code>screen</code> =		= Detaching from a running job with <code>screen</code> =

SLURM

GiulioFontana — Tue, 26 May 2026 11:09:33 GMT

Limits on jobs by the same user

← Older revision		Revision as of 11:09, 26 May 2026
(24 intermediate revisions by the same user not shown)
Line 17:		Line 17:
	;: '''the greater the fraction of Mufasa's overall resources that a job asks for, the lower the job's priority will be'''.		;: '''the greater the fraction of Mufasa's overall resources that a job asks for, the lower the job's priority will be'''.

	The priority mechanism is used to encourage users to use Mufasa's resources in an effective and equitable manner. This page includes a [[#How_to_maximise_the_priority_of_your_jobs\|chart explaining how to maximise the priority of your jobs]].		The priority mechanism is used to encourage users to use Mufasa's resources (i.e.: GPUs, CPUs, RAM, execution time) in an effective and equitable manner. This page includes a [[#How_to_maximise_the_priority_of_your_jobs\|chart explaining how to maximise the priority of your jobs]].

	The '''time''' available to a job for its execution is controlled by SLURM. When a user requests execution of a job, they must specify the duration of the time slot that the job needs. The job must complete its execution before the end of the requested time slot, ~~otherwise~~ it gets killed by SLURM.		The '''time''' available to a job for its execution is controlled by SLURM. When a user requests execution of a job, they must specify the duration of the time slot that the job needs. The job must complete its execution before the end of the requested time slot, because at the end of such slot it gets killed by SLURM.

	In Mufasa 2.0 access to system resources is managed via SLURM's ~~'''~~[[#SLURM Quality of Service (QOS)\|Quality of Service (QOS)]]~~'''~~ mechanism (Mufasa 1.0 used [[#SLURM_partitions\|partitions]] instead). ~~To launch~~ a processing job via SLURM, the user must always specify the chosen QOS. QOSes differ in the set of resources that they provide access to because each of them is designed to fit a given type of job.		In Mufasa 2.0 access to system resources is managed via SLURM's [[#SLURM Quality of Service (QOS)\|Quality of Service (QOS)]] mechanism (Mufasa 1.0 used [[#SLURM_partitions\|partitions]] instead). While [[User Jobs#Running jobs with SLURM\|launching a processing job via SLURM]], the user must always specify the chosen QOS. QOSes differ in the set of resources that they provide access to because each of them is designed to fit a given type of job.

	Mufasa sets limits to the number of jobs by the same user. This page includes a [[#Limits on jobs by the same user\|table summarising such limits]].		Mufasa sets limits to the number of jobs by the same user. This page includes a [[#Limits on jobs by the same user\|table summarising such limits]].
Line 92:		Line 92:
	! Max</br>Submit		! Max</br>Submit
	! MaxWall</br>[h]		! MaxWall</br>[h]
	! max #~~</br>~~of</br>CPUs		! max # of</br>CPUs
	! max~~</br>~~RAM</br>[GB]		! max RAM</br>[GB]
	! ~~GPU:</br>~~max # of</br>3g.20GB		! max # of</br>3g.20GB</br>GPUs
	~~! GPU:~~</br>max # of</br>4g.20GB		! max # of</br>4g.20GB</br>GPUs
	~~! GPU:~~</br>max # of</br>40GB		! max # of</br>40GB</br>GPUs
	! MinTRES</br>(GPUs)		! MinTRES</br>(GPUs)
	\|-		\|-
Line 179:		Line 179:
	A key piece of information in the table above is the '''priority tier''' associated to each QOS.		A key piece of information in the table above is the '''priority tier''' associated to each QOS.

	In Mufasa 2.0, priority tiers are used to encourage users to use the '''least powerful QOS that is compatible with their needs''', where "powerful" means "rich with resources". Less powerful QOSes increase the priority of the jobs that use them, so these jobs tend to be executed sooner.		In Mufasa 2.0, priority tiers are used to encourage users to use the '''least powerful QOS that is compatible with their needs''', where "powerful" means "rich with resources". Less powerful QOSes increase the priority of the jobs that use them, so these jobs tend to be executed sooner. See [[#Job priority\|Job priority]] for details about how priority affects the execution order of jobs in Mufasa 2.0.

	See [[#Job priority\|Job priority]] for details about how priority affects the execution order of jobs in Mufasa 2.0.

	== The <code>build</code> QOS ==		== The <code>build</code> QOS ==

	This QOS is specifically designed to be used by Mufasa users to '''build [[System#Containers\|container images]]'''. Its associated priority tier is very high, ~~so SLURM~~ jobs ~~launched~~ using this QOS ~~are~~ executed ~~quickly~~.		This QOS is specifically designed to be used by Mufasa users to '''quickly build [[System#Containers\|container images]]'''. Its associated priority tier is very high: this, combined with the fact that jobs using this QOS require few resources, means that such jobs usually get executed very soon.

	~~The~~ <code>build</code> QOS~~, though, has resources that are strictly limited to those needed for building operations; additionally, it has no access to GPUs and a short maximum duration for jobs. This makes it~~ unsuitable for ~~computing activities different from building containers~~.		On the other hand, the limited resources and the lack of access to the GPUs make the <code>build</code> QOS unsuitable for other tasks.

	See [[Singularity#Building Singularity images\|Building Singularity images]] for directions about building Singularity container images.		See [[Singularity#Building Singularity images\|Building Singularity images]] for directions about building Singularity container images.
Line 204:		Line 202:

	:: '''<code>research</code>''', i.e. academic personnel and Ph.D. students		:: '''<code>research</code>''', i.e. academic personnel and Ph.D. students
	::: * can use all [[#~~SLURM Quality of Service (QOS)~~\|~~QOSes~~]]		::: * can use all QOSes, including the the [[#Restricted QOSes\|restricted ones]]
	::: * their jobs have a higher ''base priority''		::: * their jobs have a higher ''base priority''
	::: * ~~can~~ have a higher number of jobs running at the same time		::: * have a higher number of jobs that can be running at the same time

	:: '''<code>students</code>''', i.e. M.Sc. students		:: '''<code>students</code>''', i.e. M.Sc. students
	::: * cannot use the [[#Restricted QOSes\|restricted QOSes]]		::: * cannot use the [[#Restricted QOSes\|restricted QOSes]]
	::: * their jobs have a lower ''base priority''		::: * their jobs have a lower ''base priority''
	::: * ~~can~~ have a lower number of jobs running at the same time		::: * have a lower number of jobs that can be running at the same time

	You can inspect the differences in priority and running jobs between <code>research</code> and <code>students</code> users with command		You can inspect the differences in priority and running jobs between <code>research</code> and <code>students</code> users with command
Line 249:		Line 247:
	* '''running jobs''', i.e. jobs that are currently in execution		* '''running jobs''', i.e. jobs that are currently in execution

	The following table summarises ~~the~~ limits ~~that Mufasa sets on the number of jobs by the same user~~:		Limits on submitted jobs are set via [[#SLURM Quality of Service (QOS)\|QOSes]], while limits on running jobs are set via [[#research users and students users\|user category]].

			The following table summarises what limits exist:

	{\| class="wikitable" style="text-align:center;"		{\| class="wikitable" style="text-align:center;"
	\|-		\|-
	!		!
	! on the number of <u>running</u> jobs		! on the number of <u>running</u> jobs</br>by a single user
	! on the number of <u>submitted</u> jobs		! on the number of <u>submitted</u> jobs</br>by a single user
	\|-		\|-
	! rowspan="1" style="text-align:center;" \| global limits<br/>(system-wide)		! rowspan="1" style="text-align:center;" \| global limits<br/>(system-wide)
	\| '''''2 for'' <code>research</code> ''users'''''<br/>'''''1 for'' <code>students</code> ''users'''''~~<br/>[limits explained [[#research users and students users\|here]]]~~		\| '''''2 for'' <code>research</code> ''users'''''<br/>'''''1 for'' <code>students</code> ''users'''''
	\| '''''not limited directly...'''''<br/>...but subject to limits set by individual QOSes (below)		\| '''''not limited directly...'''''<br/>...but subject to limits set by individual QOSes (below)
	\|-		\|-
	! rowspan="1" style="text-align:center;" \| limits for<br/>individual QOSes		! rowspan="1" style="text-align:center;" \| limits for<br/>individual</br>QOSes
	\| '''''not limited directly...'''''<br/>...but subject to global limits on running jobs (above)<br/>and to limits set by individual QOSes (on the right)		\| '''''not limited directly...'''''<br/>...but subject to global limits on running jobs (above)<br/>and to limits set by individual QOSes (on the right)
	\| '''''2 for the'' <code>gpuwide</code> ''QOS'''''<br/>'''''1 for each of the other QOSes'''''~~<br/>[limits explained [[#SLURM Quality of Service (QOS)\|here]]]~~		\| '''''2 for the'' <code>gpuwide</code> ''QOS'''''<br/>'''''1 for each of the other QOSes'''''
	\|}		\|}