Appendix I — SLURM Command Reference

This appendix is a compact reference for SLURM (Simple Linux Utility for Resource Management) (Yoo et al., 2003) commands and directives on clusters like Talapas. The concepts behind them (fair-share scheduling, right-sizing, pipeline design) are explained in Chapter 20.

I.1 The Ten Commands You Will Actually Use

Table I.1: The SLURM commands used day to day
Task Command
Submit a batch script sbatch script.sh
Submit with a one-off override sbatch --mem=64G --time=08:00:00 script.sh
Interactive shell on a compute node srun --account=myPIRG --partition=interactive --time=2:00:00 --cpus-per-task=2 --mem=8G --pty bash
List your jobs (with pending reason) squeue -u $USER
Estimated start time of a pending job squeue -j JOBID --start
Cancel a job / all your pending jobs scancel JOBID / scancel -u $USER -t PENDING
Efficiency report for a finished job seff JOBID
Accounting fields for a finished job sacct -j JOBID --format=JobID,ReqMem,MaxRSS,Elapsed,State,ExitCode
Everything SLURM knows about a job scontrol show job JOBID
Partition and node availability sinfo

I.2 Job Submission

Table I.2: Job submission commands
Command Description Example
sbatch Submit batch job sbatch script.sh
sbatch --parsable Return only job ID sbatch --parsable script.sh
srun Run interactive command srun --pty bash
srun --pty Allocate pseudo-terminal srun --pty -p interactive bash
salloc Allocate resources salloc -N 1 -t 1:00:00

I.3 Job Management

Table I.3: Job management commands
Command Description Example
squeue Show job queue squeue
squeue -u $USER Show your jobs squeue -u $USER
squeue -p partition Show partition jobs squeue -p compute
squeue -j JOBID Show specific job squeue -j 123456
squeue -l Long format squeue -l
squeue --start Show start times squeue --start
scancel JOBID Cancel specific job scancel 123456
scancel -u $USER Cancel all your jobs scancel -u $USER
scancel -n NAME Cancel by job name scancel -n analysis
scancel -p PARTITION Cancel by partition scancel -p compute
scontrol hold JOBID Hold job scontrol hold 123456
scontrol release JOBID Release held job scontrol release 123456

I.4 Job Information

Table I.4: Job information commands
Command Description Example
scontrol show job JOBID Detailed job info scontrol show job 123456
scontrol show node NODE Node information scontrol show node n001
sacct Job accounting sacct
sacct -j JOBID Accounting for job sacct -j 123456
sacct --format Custom format sacct --format=JobID,Elapsed,MaxRSS
sacct -S YYYY-MM-DD Jobs since date sacct -S 2024-01-01
seff JOBID Job efficiency report seff 123456

Useful sacct Format Fields

Table I.5: sacct format fields
Field Description
JobID Job identifier
JobName Job name
Partition Partition used
Account Account charged
State Job state
ExitCode Exit code
Elapsed Wall time used
TotalCPU CPU time used
MaxRSS Max memory used
MaxVMSize Max virtual memory
ReqMem Requested memory
ReqCPUS Requested CPUs
AllocCPUS Allocated CPUs
NodeList Nodes used
Start Start time
End End time

Reading seff and Right-Sizing

Run seff JOBID after every job. The loop: first run with a modest over-estimate; then set the next run from what was actually used.

Table I.6: Turning seff output into the next job’s requests
seff line Use it to set
State: COMPLETED (else FAILED, OUT_OF_MEMORY, TIMEOUT, CANCELLED) Whether to trust the output at all; read the .err log if not COMPLETED
Memory Utilized (= MaxRSS in sacct) Next --mem = this × 1.2
Memory Efficiency Under 50%? Shrink --mem
Job Wall-clock time (= Elapsed) Next --time = this × 1.2
CPU Efficiency Low on a multi-core request means the tool is not using the cores; reduce --cpus-per-task

Defaults on Talapas if you do not specify: about 3.7 GB memory per core, 1 core, and the partition’s maximum time (24 h on compute). There is no default partition or account; both are required.

Why Is My Job Pending?

Shown in the NODELIST(REASON) column of squeue -u $USER.

Table I.7: Common pending reasons
Reason Meaning Action
Priority Higher fair-share priority jobs are ahead Wait
Resources Requested resources not free yet Wait; check requests are not oversized
QOSMaxCpuPerUserLimit / QOSMaxJobsPerUserLimit / QOSMaxMemoryPerUser Per-user cap on cores / jobs / memory reached Wait for your own jobs to finish; throttle arrays with %M
ReqNodeNotAvail, Reserved for maintenance --time runs into a maintenance window Shorten --time or add --time-min
PartitionTimeLimit --time exceeds the partition maximum Shorten, or use the *long partition
AssocGrpCPUMinutesLimit PIRG allocation exhausted Contact RACS
Dependency Waiting on --dependency= job Normal
InvalidAccount --account missing or wrong groups; fix --account=

I.5 Cluster Information

Table I.8: Cluster information commands
Command Description Example
sinfo Partition overview sinfo
sinfo -p PARTITION Specific partition sinfo -p compute
sinfo -N Node-oriented view sinfo -N
sinfo -l Long format sinfo -l
sinfo --states=idle Nodes in state sinfo --states=idle
scontrol show partition Partition details scontrol show partition compute
scontrol show config SLURM configuration scontrol show config
sprio Job priority factors sprio
sshare Share information sshare -u $USER

Node States

Table I.9: Node states
State Description
idle Available, no jobs
alloc Fully allocated
mix Partially allocated
down Unavailable
drain Not accepting new jobs
maint Under maintenance
resv Reserved

Job States

Table I.10: Job states
State Description
PD Pending (waiting to run)
R Running
CG Completing
CD Completed
F Failed
TO Timeout
CA Cancelled
NF Node failure
OOM Out of memory

I.6 SBATCH Directives

Essential Directives

#!/bin/bash
#SBATCH --job-name=my_job       # Job name (displayed in queue)
#SBATCH --partition=compute     # Partition to use (required)
#SBATCH --account=PIRG          # Account to charge (required)
#SBATCH --time=04:00:00         # Time limit (HH:MM:SS or D-HH:MM:SS)
#SBATCH --nodes=1               # Number of nodes
#SBATCH --ntasks=1              # Number of tasks (MPI ranks)
#SBATCH --cpus-per-task=4       # CPUs per task (threads)
#SBATCH --mem=16G               # Memory per node

set -euo pipefail               # stop on error / unset var / failed pipe stage
module purge                    # modules must be loaded INSIDE the script
module load R/4.3.3

Output Directives

#SBATCH --output=output_%j.log  # Standard output (%j = job ID)
#SBATCH --error=error_%j.log    # Standard error
#SBATCH --output=%x_%j.out      # %x = job name
#SBATCH --mail-user=email@uoregon.edu  # Email address
#SBATCH --mail-type=BEGIN,END,FAIL     # When to email

Resource Directives

#SBATCH --mem-per-cpu=4G        # Memory per CPU
#SBATCH --gres=gpu:1            # Request 1 GPU
#SBATCH --gres=gpu:v100:2       # Request 2 V100 GPUs
#SBATCH --constraint=intel      # Node constraint
#SBATCH --exclusive             # Exclusive node access
#SBATCH --array=1-100           # Job array
#SBATCH --array=1-100%10        # Array with max 10 running

Dependency Directives

#SBATCH --dependency=afterok:JOBID      # Run after job succeeds
#SBATCH --dependency=afterany:JOBID     # Run after job completes
#SBATCH --dependency=afternotok:JOBID   # Run after job fails
#SBATCH --dependency=singleton          # Only one job with this name

I.7 Complete SBATCH Directive Reference

Table I.11: Complete SBATCH directives
Directive Description Example
--job-name / -J Job name --job-name=analysis
--partition / -p Partition --partition=compute
--account / -A Account --account=myPIRG
--time / -t Time limit; exceeded → TIMEOUT --time=04:00:00
--time-min Minimum acceptable time (lets backfill start the job in a shorter gap) --time-min=01:00:00
--nodes / -N Number of nodes --nodes=2
--ntasks / -n Number of tasks --ntasks=8
--ntasks-per-node Tasks per node --ntasks-per-node=4
--cpus-per-task / -c CPUs per task --cpus-per-task=4
--mem Memory per node; exceeded → OUT_OF_MEMORY --mem=32G
--mem-per-cpu Memory per CPU --mem-per-cpu=4G
--gres Generic resources --gres=gpu:1
--output / -o Stdout file (%j job ID, %x name, %A_%a array) --output=logs/%x_%j.out
--error / -e Stderr file (omit to merge into --output) --error=logs/%x_%j.err
--mail-user Email address --mail-user=me@uoregon.edu
--mail-type Email events --mail-type=ALL
--array Job array (%M = max concurrent) --array=1-100%10
--dependency Job dependencies --dependency=afterok:123
--constraint Node features --constraint=intel
--exclusive Exclusive access --exclusive
--nodelist Specific nodes --nodelist=n[001-004]
--exclude Exclude nodes --exclude=n001
--qos Quality of service --qos=long
--reservation Use reservation --reservation=workshop

Output Filename Patterns

Table I.12: Filename patterns
Pattern Replaced With
%j Job ID
%x Job name
%N Short hostname
%n Node identifier
%A Array job ID
%a Array task ID
%u Username

I.8 Module Commands

Table I.13: Module commands
Command Description Example
module avail List available modules module avail
module avail NAME Search modules module avail python
module spider NAME Detailed search module spider R
module load NAME Load module module load R/4.3.3
module load NAME/VER Load specific version module load miniconda3/20240410
module unload NAME Unload module module unload R
module list Show loaded modules module list
module purge Unload all modules module purge
module show NAME Show module details module show R/4.3.3
module help NAME Module help module help python
module swap OLD NEW Replace module module swap gcc/9 gcc/11
module save NAME Save collection module save mymodules
module restore NAME Restore collection module restore mymodules
module savelist List collections module savelist

I.9 Job Arrays

Template: One Script, Many Samples

#!/bin/bash
#SBATCH --job-name=align
#SBATCH --partition=compute
#SBATCH --account=myPIRG
#SBATCH --time=02:00:00
#SBATCH --cpus-per-task=8
#SBATCH --mem=16G
#SBATCH --array=1-24%8               # 24 tasks, at most 8 running at once
#SBATCH --output=logs/%x_%A_%a.out   # %A = array job ID, %a = task ID
#SBATCH --error=logs/%x_%A_%a.err

set -euo pipefail
module purge
module load bwa/0.7.17 samtools/1.17

SAMPLE=$(sed -n "${SLURM_ARRAY_TASK_ID}p" samples.txt)   # line N = task N
OUT="results/${SAMPLE}.bam"
[[ -s "$OUT" ]] && { echo "$SAMPLE done, skipping"; exit 0; }   # resumable

bwa mem -t "$SLURM_CPUS_PER_TASK" ref.fa "raw/${SAMPLE}_R1.fastq.gz" "raw/${SAMPLE}_R2.fastq.gz" \
    | samtools sort -@ "$SLURM_CPUS_PER_TASK" -o "$OUT.tmp"
mv "$OUT.tmp" "$OUT"          # rename only after success: no truncated outputs
echo "Wrote $OUT"

Run mkdir -p logs results before submitting. The logs/ directory must exist or the job fails silently.

Array Index Variants

#SBATCH --array=0-100:10        # 0, 10, 20, ..., 100
#SBATCH --array=1-1000%50       # Max 50 running at once
#SBATCH --array=3,7,12          # Rerun only specific tasks

To size the array from a file at submit time: sbatch --array=1-$(wc -l < samples.txt) align.sh. To run a step only after every task succeeds: sbatch --dependency=afterok:ARRAY_JOB_ID next_step.sh.

I.10 Environment Variables

Table I.14: SLURM environment variables
Variable Description
$SLURM_JOB_ID Job ID
$SLURM_JOB_NAME Job name
$SLURM_SUBMIT_DIR Directory where job was submitted
$SLURM_JOB_NODELIST List of nodes
$SLURM_NNODES Number of nodes
$SLURM_NTASKS Number of tasks
$SLURM_CPUS_PER_TASK CPUs per task
$SLURM_MEM_PER_NODE Memory per node
$SLURM_ARRAY_TASK_ID Array task ID
$SLURM_ARRAY_JOB_ID Array job ID
$SLURM_TMPDIR Temporary directory
$SLURM_NODEID Node ID within job
$SLURM_PROCID Task ID within job
$SLURM_LOCALID Local task ID on node

I.11 Example Job Scripts

Basic R Script

#!/bin/bash
#SBATCH --job-name=r_analysis
#SBATCH --partition=compute
#SBATCH --account=myPIRG
#SBATCH --time=02:00:00
#SBATCH --nodes=1
#SBATCH --cpus-per-task=4
#SBATCH --mem=16G
#SBATCH --output=r_job_%j.log

set -euo pipefail

# Load required modules (inside the script, with a version number)
module purge
module load R/4.3.3

# Run R script
Rscript my_analysis.R

Python with Conda

#!/bin/bash
#SBATCH --job-name=python_job
#SBATCH --partition=compute
#SBATCH --account=myPIRG
#SBATCH --time=04:00:00
#SBATCH --cpus-per-task=8
#SBATCH --mem=32G
#SBATCH --output=python_%j.log

# Load conda and activate your environment
module load miniconda3/20240410
conda activate myenv

# Run Python script
python analysis.py --input data.csv --output results/

GPU Job

#!/bin/bash
#SBATCH --job-name=gpu_job
#SBATCH --partition=gpu
#SBATCH --account=myPIRG
#SBATCH --time=08:00:00
#SBATCH --nodes=1
#SBATCH --cpus-per-task=4
#SBATCH --mem=32G
#SBATCH --gres=gpu:1
#SBATCH --output=gpu_%j.log

module load cuda/12.4
module load miniconda3/20240410
conda activate ml_env

python train_model.py

MPI Job

#!/bin/bash
#SBATCH --job-name=mpi_job
#SBATCH --partition=compute
#SBATCH --account=myPIRG
#SBATCH --time=04:00:00
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=16
#SBATCH --mem-per-cpu=4G
#SBATCH --output=mpi_%j.log

module load openmpi/4.1

mpirun -np 64 ./my_mpi_program

I.12 Talapas-Specific Information

Common Partitions

Table I.15: Talapas partitions
Partition Description Time Limit
compute Standard CPU nodes 1 day
computelong Extended-time CPU jobs 14 days
gpu GPU nodes 1 day
gpulong Extended-time GPU jobs 14 days
interactive Interactive CPU sessions 12 hours
interactivegpu Interactive GPU sessions 8 hours
memory High-memory nodes (up to 4 TB) 1 day
memorylong Extended-time high-memory jobs 14 days

Run sinfo to list every partition, including your lab’s condo partitions. Interactive sessions should use --partition=interactive (or interactivegpu).

Talapas Helper Commands

Table I.16: Talapas-specific helper commands (see Section 20.4 for access details)
Task Command
Which PIRGs (accounts) am I in? groups
Which GPUs exist and which are free? /packages/racs/bin/slurm-show-gpus
Set my default account once echo 'export SBATCH_ACCOUNT=myPIRG' >> ~/.bash_profile
Home and project usage df -h ~ / du -sh /projects/myPIRG/$USER

Storage Locations

Table I.17: Talapas storage. Nothing is backed up by RACS; use Globus for large transfers (Section 20.5.3).
Location Quota Purpose
/home/$USER 250 GB Config files, small data
/projects/PIRG/$USER Shared (2 TB default) Large data, analysis
/projects/PIRG/shared Shared Shared group data
/scratch/PIRG Large per-PIRG quota Working data; purged after 90 days without access
$SLURM_TMPDIR Local Temporary scratch