Welcome to Computer Center

Menu

SLURM batch system

Important commands

sbatch: Submitting batch jobs

The command sbatch submits your batch jobs to the cluster. Hardware ressources are requested via #SBATCH directives.
  • Please take care that you request exactly the ressources which your script later needs.
      • If you request 8 CPU cores and your application run serially, then 7 cores remain unused.
      • If you request only one core and your applcation wants to parallelise with 8 cores, you will slow down other users or yourself on that compute node.
  • You can be more generous with the walltime. If you have requested 100 hours and your jobs needs just 80 hours, the hardware will be released immediately for the next job.
  • The cluster is seperated into several partitions. Your job will be assigned a partition depending on your hardware requirements.

SBATCH Option example remark
--ntasks --ntasks 32 Number of MPI processes.
--cpus-per-task --cpus-per-task 8 defines the number of CPU cores per task.
--mem-per-cpu --mem-per-cpu 2000 requests memory (in MB per CPU core).
--time --time 4:00:00 makes a reservation for the given time.
--gres --gres gpu:1 request a generic hardware ressources (here: a GPU).
Specific GPU models can be chosen by:
--gres gpu:a100:1
--gres gpu:h100:1
--mail-type [BEGIN|END|FAIL|REQUEUE|ALL] --mail-type FAIL defines if and when to send an e-mail to notify the user about certain events.
--mail-user --mail-user name@tuhh.de TUHH E-mail address for notifications. External E-mail addresses are not supported.
--time-min --time-min 1:00:00 SLURM starts the job even if there is less computing time available the requested (--time) but at least as much as passed here. Useful for applications which may stop anytime. Leads often to shorter waiting times.
--test-only --test-only Does not submit the job, but checks the syntax and guesses when and where the job would start.

squeue: show jobs

The command squeue list all pending and running jobs. The documentation for squeue lists versatile options for output formatting, especially expected start time and priority.

scancel: cancel jobs

scancel deletes jobs.

sview: graphical front end

All tasks can be accomplished with the graphical front end sview.

Environment variables in SLURM

You can access SLURM related values inside your batch scripts which have not been known when submitting the jobs with SLURM specific environment variables such as the Job ID.
SLURM offers various environment variables (cf. manpage for srun), such as $SLURM_JOB_ID.

Environment modules

The HPC cluster has a set of common preinstalled scientific applications. Moreover every user can install user-specific software into the home directory.

Most preinstalled software applications and development tools are accessible via the environment modules. A summary of the installed software is displayed with the command module avail.
The subcommand module load initializes a specific software, i.e. environment variables are set to easily access the chosen software. In case of Matlab, module load matlab/2022b allows to start Matlab 2022b directly with the command matlab.
module unload withdraws the modification from a certain software module.

Python can be used via the Anaconda environment module or via the system-side installed Python (/usr/bin/python3).

The CFD software OpenFOAM can be initialized by starting an application specific script and is described here .

Example scripts

SMP parallel job

The following script is to perform an Abaqus computation. FEM application usually do not scale well and tend to use intensive I/O, so we request 8 cores on a single machine. To avoid performance degradation due to intensive I/O, we use the local storage below /usertemp.
Moreover, we request 5GB RAM per core, i.e. 40GB RAM in total. 

#!/bin/bash -l
#SBATCH --ntasks 1
#SBATCH --cpus-per-task 8
#SBATCH --mem-per-cpu 5000
#SBATCH --time=12:00:00

# initialize module system and load Abaqus module 
. /etc/profile.d/module.sh
module load abaqus/2021

# Create working directory below $SCRATCH
MYWORKDIR=$SCRATCH/$SLURM_JOBID
mkdir $MYWORKDIR

# Copy input file from the submitting directory to our working directory
cp $SLURM_SUBMIT_DIR/Beispiel.inp $MYWORKDIR

# change to working directory and start computation
cd $MYWORKDIR
abaqus job=Beispiel input=Beispiel cpus=8 interactive

# copy back important results
cp -r Beispiel.* $SLURM_SUBMIT_DIR

# Clean up
rm -rf $MYWORKDIR

exit
Consider that $SCRATCH on the login nodes and the compute nodes are local (i.e. different) directories.

Therefore the input files are first copied during job runtime from a network file system to the SSD and vice-versa after finishing the computation.

MPP parallel job

The following script is to perform a generic MPI-parallel computation with the software my_program. As the software runs on multiple nodes we use the network storage below /work. 

#!/bin/bash -l
#SBATCH --ntasks 32
#SBATCH --cpus-per-task 1
#SBATCH --mem-per-cpu 2000
#SBATCH --time=12:00:00

# initialize module system and load module for Intel compiler and MPI
. /etc/profile.d/module.sh
module load intel/2019

# Create working directory below $WORK
MYWORKDIR=$WORK/$SLURM_JOBID
mkdir $MYWORKDIR

# Copy input file from the submitting directory to our working directory
cp ~/my_case/inputfile $MYWORKDIR
cd $MYWORKDIR

# Create host file with the information how to distribute the workload
srun hostname > hostfile_$SLURM_JOBID

# start computation
mpirun -np 32 -machinefile hostfile_$SLURM_JOBID my_program

# copy output files back
cp -r * ~/my_case

# Clean up
rm -rf $MYWORKDIR

exit

GPU-Job

The following script is to perform a computation with the software gromacs on 8 CPU cores and one GPU card. The environment variable GPU_DEVICE_ORDINAL tells us the ID(s) of the assigned GPUs. 

#!/bin/bash -l
#SBATCH --ntasks 1
#SBATCH --cpus-per-task 8
#SBATCH --gres gpu:1
#SBATCH --mem-per-cpu 2000
#SBATCH --time 12:00:00

# initialize module system and load Abaqus module
. /etc/profile.d/module.sh
module load gromacs/4.6.5-gpu

# Create working directory below $SCRATCH
MYWORKDIR=$SCRATCH/$SLURM_JOBID
mkdir $MYWORKDIR

# Copy input files from the submitting directory to our working directory
cp ~/gromacs_data/* $MYWORKDIR

# change to working directory and start computation
cd $MYWORKDIR
mdrun_mpi -ntomp 8 -v -s a.tpr -o a.trr -c a.pdb -e a.edr -g a.log

# Copy results back
cp * ~/gromacs_data/

# Clean up
rm -rf $MYWORKDIR

exit